2024-09-16T11:12:05.585 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T11:12:05.605 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T11:12:05.703 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} email: sangadi@redhat.com first_in_suite: false job_id: '7907185' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon name: sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd objectstore: filestore osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOe7lMxi92VYCpCtte6dzjkYKO5OGVDJ7IZhNUiqKzRWoeCIfgcJG4poQUlmzBExppekzihp9JzY978AaCEVF+Q= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMCaz0LRwVAGDTbGa15b9GXxNEYnUK+l3jpCRHQyjqf7CNnXJKrdfB7vxJ/LVkYxdAuKcPrxoCK843457LUB6I= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_journal.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_10:44:09 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T11:12:05.703 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-16T11:12:05.704 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-16T11:12:05.704 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T11:12:05.705 INFO:teuthology.task.internal:Checking packages... 2024-09-16T11:12:05.727 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-16T11:12:05.727 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T11:12:05.727 INFO:teuthology.packaging:ref: None 2024-09-16T11:12:05.727 INFO:teuthology.packaging:tag: None 2024-09-16T11:12:05.727 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T11:12:05.727 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:12:05.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T11:12:05.886 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-16T11:12:05.889 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T11:12:05.895 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T11:12:05.895 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T11:12:05.912 INFO:teuthology.task.internal:Saving configuration 2024-09-16T11:12:05.930 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T11:12:05.938 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T11:12:05.965 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 11:07:01.142485', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOe7lMxi92VYCpCtte6dzjkYKO5OGVDJ7IZhNUiqKzRWoeCIfgcJG4poQUlmzBExppekzihp9JzY978AaCEVF+Q='} 2024-09-16T11:12:05.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 11:07:01.141065', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMCaz0LRwVAGDTbGa15b9GXxNEYnUK+l3jpCRHQyjqf7CNnXJKrdfB7vxJ/LVkYxdAuKcPrxoCK843457LUB6I='} 2024-09-16T11:12:05.989 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T11:12:05.995 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-16T11:12:05.995 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-16T11:12:05.995 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T11:12:06.057 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fde037a3ac0>, signals=[15]) 2024-09-16T11:12:06.057 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T11:12:06.062 INFO:teuthology.task.internal:Opening connections... 2024-09-16T11:12:06.062 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:12:06.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:12:06.140 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:12:06.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:12:06.217 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T11:12:06.225 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-09-16T11:12:06.244 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-09-16T11:12:06.245 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-09-16T11:12:06.302 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-09-16T11:12:06.302 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-09-16T11:12:06.302 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-09-16T11:12:06.302 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-09-16T11:12:06.302 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-09-16T11:12:06.302 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T11:12:06.303 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T11:12:06.304 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-09-16T11:12:06.326 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-09-16T11:12:06.345 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-09-16T11:12:06.345 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-09-16T11:12:06.404 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T11:12:06.405 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-09-16T11:12:06.405 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T11:12:06.405 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T11:12:06.405 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T11:12:06.405 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-09-16T11:12:06.426 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T11:12:06.436 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T11:12:06.443 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T11:12:06.443 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-09-16T11:12:06.446 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-09-16T11:12:06.463 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T11:12:06.493 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T11:12:06.493 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-09-16T11:12:06.503 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-09-16T11:12:06.601 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T11:12:06.655 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T11:12:06.671 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T11:12:06.671 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T11:12:06.672 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T11:12:06.672 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T11:12:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:12:06.689 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-09-16T11:12:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:12:06.690 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-16T11:12:06.747 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-503.el9.x86_64 2024-09-16T11:12:06.748 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-503.el9.x86_64 2024-09-16T11:12:06.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-09-16T11:12:06.749 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-503.el9.x86_64 2024-09-16T11:12:06.750 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-503.el9.x86_64 2024-09-16T11:12:06.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-09-16T11:12:10.646 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.2 MB 00:02 2024-09-16T11:12:10.652 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.2 MB 00:02 2024-09-16T11:12:12.561 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-16T11:12:18.045 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T11:12:19.630 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-16T11:12:20.817 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-16T11:12:26.606 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 4.5 kB/s | 1.7 kB 00:00 2024-09-16T11:12:26.861 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 20 MB 00:15 2024-09-16T11:12:28.629 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T11:12:28.629 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T11:12:28.715 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T11:12:28.721 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T11:12:28.722 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T11:12:28.722 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:12:28.722 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:12:28.722 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:12:28.722 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-16T11:12:28.722 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:12:28.723 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 79 M 2024-09-16T11:12:28.723 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 126 M 2024-09-16T11:12:28.723 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:12:29.033 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 131 kB/s | 23 kB 00:00 2024-09-16T11:12:32.787 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.5 MB 00:01 2024-09-16T11:12:33.041 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-09-16T11:12:34.571 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-16T11:12:35.392 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-16T11:12:36.257 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.7 MB/s | 35 MB 00:07 2024-09-16T11:12:37.786 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.3 MB/s | 29 MB 00:08 2024-09-16T11:12:37.787 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:12:37.787 INFO:teuthology.orchestra.run.smithi204.stdout:Total 8.7 MB/s | 79 MB 00:09 2024-09-16T11:12:38.476 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:12:38.834 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:12:38.834 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:12:39.834 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:12:39.834 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:12:41.056 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T11:12:42.283 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:12:43.113 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T11:12:43.113 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T11:12:43.134 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:12:43.223 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:12:43.227 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:12:43.228 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-09-16T11:12:43.229 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:12:43.229 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 79 M 2024-09-16T11:12:43.229 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 126 M 2024-09-16T11:12:43.229 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:12:43.234 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:12:43.399 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 187 kB/s | 23 kB 00:00 2024-09-16T11:12:46.039 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:12:46.172 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:12:50.099 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 2.3 MB/s | 16 MB 00:06 2024-09-16T11:12:50.988 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:12:51.031 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:12:55.931 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:12:57.048 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 2.1 MB/s | 29 MB 00:13 2024-09-16T11:12:59.182 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-16T11:12:59.182 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:12:59.182 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.9 MB/s | 79 MB 00:15 2024-09-16T11:12:59.896 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:13:00.298 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:13:00.298 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:13:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:13:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:13:03.828 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:13:04.491 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:13:04.582 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:13:07.368 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:13:07.495 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:13:12.621 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:13:12.663 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:17.807 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:34.130 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:34.169 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:34.522 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:13:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:13:34.523 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:13:35.332 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:35.332 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:13:35.332 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:13:35.332 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T11:13:35.332 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T11:13:35.333 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T11:13:35.333 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T11:13:35.333 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:13:35.333 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:13:35.660 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:13:35.752 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:13:35.753 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:13:35.753 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T11:13:35.753 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T11:13:35.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T11:13:35.753 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-16T11:13:35.753 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T11:13:35.753 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:13:35.812 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:13:35.812 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:13:35.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 82.el9 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8274670 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T11:13:35.852 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:13:35.853 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T11:13:35.854 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T11:13:35.854 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-505.el9.x86_64 2024-09-16T11:13:35.855 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T11:13:36.475 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-09-16T11:13:37.834 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T11:13:37.864 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-09-16T11:13:37.868 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-16T11:13:37.883 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.kwN3iohuqO 2024-09-16T11:13:37.883 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kwN3iohuqO 2024-09-16T11:13:37.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.kwN3iohuqO 2024-09-16T11:13:38.092 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.kwN3iohuqO is 6KB 2024-09-16T11:13:38.141 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.kwN3iohuqO 2024-09-16T11:13:38.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T11:13:38.227 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T11:13:38.227 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T11:13:38.227 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T11:13:38.227 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T11:13:38.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T11:13:38.384 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-09-16T11:13:56.635 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:56.677 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:57.161 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:57.161 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T11:13:57.162 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T11:13:57.162 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T11:13:58.059 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T11:13:58.059 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:13:58.059 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:13:58.059 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T11:13:58.060 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T11:13:58.060 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T11:13:58.060 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T11:13:58.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:13:58.060 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:13:58.376 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:13:58.509 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:13:58.510 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:13:58.510 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T11:13:58.510 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T11:13:58.510 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T11:13:58.510 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-16T11:13:58.510 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T11:13:58.511 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T11:13:58.569 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T11:13:58.570 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T11:13:58.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-09-16T11:13:58.601 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-09-16T11:13:58.601 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-09-16T11:13:58.601 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 82.el9 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8274670 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-09-16T11:13:58.602 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:13:58.603 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T11:13:58.604 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T11:13:58.604 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-505.el9.x86_64 2024-09-16T11:13:58.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T11:13:59.309 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-09-16T11:14:00.668 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T11:14:00.696 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-09-16T11:14:00.698 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-09-16T11:14:00.713 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.vLRDGHSBiP 2024-09-16T11:14:00.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vLRDGHSBiP 2024-09-16T11:14:00.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.vLRDGHSBiP 2024-09-16T11:14:00.923 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.vLRDGHSBiP is 6KB 2024-09-16T11:14:00.971 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.vLRDGHSBiP 2024-09-16T11:14:00.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T11:14:01.055 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T11:14:01.056 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T11:14:01.056 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T11:14:01.056 INFO:teuthology.orchestra.run.smithi053.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T11:14:01.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T11:14:01.199 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-09-16T11:14:08.393 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-16T11:14:08.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T11:14:08.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:14:31.221 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-16T11:14:31.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-16T11:14:31.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:14:42.877 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-16T11:14:51.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T11:14:51.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:15:10.273 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-16T11:15:22.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T11:15:22.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:15:22.850 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-16T11:15:23.262 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T11:15:23.262 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T11:15:23.262 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-16T11:15:23.277 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-505.el9.x86_64 2024-09-16T11:15:23.278 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T11:15:23.278 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T11:15:23.278 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T11:15:24.279 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-16T11:15:24.279 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T11:15:24.404 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-09-16T11:15:24.427 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:31.224 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T11:15:40.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-16T11:15:40.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:15:40.560 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-09-16T11:15:40.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-16T11:15:40.926 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T11:15:40.927 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-09-16T11:15:40.943 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-505.el9.x86_64 2024-09-16T11:15:40.943 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T11:15:40.943 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T11:15:40.944 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T11:15:41.945 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-16T11:15:41.945 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T11:15:42.010 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-09-16T11:15:42.026 DEBUG:teuthology.parallel:result is None 2024-09-16T11:15:42.027 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T11:15:42.034 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T11:15:42.035 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T11:15:42.037 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T11:15:42.058 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T11:15:42.066 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T11:15:42.129 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T11:15:42.130 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T11:15:42.132 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T11:15:42.168 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T11:15:42.176 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T11:15:42.176 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T11:15:42.204 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T11:15:42.236 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:42.240 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:42.248 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:42.252 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:42.254 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T11:15:42.261 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T11:15:42.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T11:15:42.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T11:15:42.325 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T11:15:42.334 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T11:15:42.334 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T11:15:42.366 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T11:15:42.383 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:15:42.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:15:42.518 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:15:42.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:15:42.648 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:15:42.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T11:15:42.716 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:15:42.755 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T11:15:42.819 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:15:42.856 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T11:15:42.902 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:15:42.902 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T11:15:42.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-09-16T11:15:42.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-09-16T11:15:43.019 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:15:43.057 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:15:43.330 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T11:15:43.337 INFO:teuthology.task.internal:Starting timer... 2024-09-16T11:15:43.337 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T11:15:43.353 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T11:15:43.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-09-16T11:15:43.414 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-09-16T11:15:43.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-09-16T11:15:43.489 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-09-16T11:15:43.491 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T11:15:43.491 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-09-16T11:15:43.562 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-09-16T11:15:43.562 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-09-16T11:15:43.591 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-09-16T11:15:43.592 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-09-16T11:15:43.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:15:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:15:43.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:15:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:15:43.657 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T11:15:43.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-09-16T11:15:43.698 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-09-16T11:15:43.726 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T11:15:43.736 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T11:15:43.747 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T11:15:43.748 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T11:19:30.360 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-09-16T11:19:30.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-16T11:19:30.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:19:30.441 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-09-16T11:19:30.521 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-09-16T11:19:30.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T11:19:30.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T11:19:30.596 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-16T11:19:30.676 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-16T11:19:30.677 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T11:19:30.688 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T11:19:30.689 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T11:19:30.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:19:30.692 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T11:19:30.692 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:19:30.720 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T11:19:30.735 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T11:19:30.746 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T11:19:30.760 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T11:19:30.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-09-16T11:19:30.772 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-09-16T11:19:30.785 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-09-16T11:19:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T11:19:30.796 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-09-16T11:19:30.802 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T11:19:30.810 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T11:19:30.824 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T11:19:30.860 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:19:30.864 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:19:30.864 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-09-16T11:19:30.864 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.864 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.864 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.864 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.881 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.886 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:19:30.887 INFO:teuthology.run_tasks:Running task install... 2024-09-16T11:19:30.898 DEBUG:teuthology.task.install:project ceph 2024-09-16T11:19:30.898 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-16T11:19:30.898 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-16T11:19:30.898 INFO:teuthology.task.install:Using flavor: default 2024-09-16T11:19:30.904 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T11:19:30.904 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-16T11:19:30.905 DEBUG:teuthology.orchestra.run.smithi053:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T11:19:30.905 DEBUG:teuthology.orchestra.run.smithi204:> sudo dnf -y install 'dnf-command(copr)' 2024-09-16T11:19:31.483 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:48 ago on Mon 16 Sep 2024 11:18:43 AM UTC. 2024-09-16T11:19:31.487 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:45 ago on Mon 16 Sep 2024 11:18:46 AM UTC. 2024-09-16T11:19:31.570 INFO:teuthology.orchestra.run.smithi204.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T11:19:31.577 INFO:teuthology.orchestra.run.smithi053.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-16T11:19:31.614 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:19:31.615 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:19:31.615 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:19:31.622 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:19:31.623 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:19:31.623 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:19:31.673 DEBUG:teuthology.orchestra.run.smithi204:> sudo dnf -y copr enable ceph/el9 2024-09-16T11:19:31.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo dnf -y copr enable ceph/el9 2024-09-16T11:19:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T11:19:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:of the main distribution, and quality may vary. 2024-09-16T11:19:31.843 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:19:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T11:19:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T11:19:31.843 INFO:teuthology.orchestra.run.smithi204.stderr:, 2024-09-16T11:19:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:and packages are not held to any quality or security level. 2024-09-16T11:19:31.844 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:19:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T11:19:31.844 INFO:teuthology.orchestra.run.smithi204.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:of the main distribution, and quality may vary. 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:, 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:and packages are not held to any quality or security level. 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-16T11:19:31.863 INFO:teuthology.orchestra.run.smithi053.stderr:Please do not file bug reports about these packages in Fedora 2024-09-16T11:19:31.864 INFO:teuthology.orchestra.run.smithi053.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-16T11:19:31.916 INFO:teuthology.orchestra.run.smithi204.stdout:Repository successfully enabled. 2024-09-16T11:19:31.919 INFO:teuthology.orchestra.run.smithi053.stdout:Repository successfully enabled. 2024-09-16T11:19:31.953 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T11:19:31.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:19:31.958 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-16T11:19:31.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:19:32.103 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T11:19:32.103 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T11:19:32.113 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-16T11:19:32.113 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-16T11:19:32.214 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T11:19:32.214 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:19:32.214 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T11:19:32.223 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T11:19:32.223 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:19:32.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T11:19:32.239 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T11:19:32.240 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T11:19:32.249 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T11:19:32.250 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T11:19:32.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T11:19:32.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T11:19:32.383 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T11:19:32.393 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T11:19:32.449 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-09-16T11:19:32.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-09-16T11:19:32.459 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-09-16T11:19:32.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-09-16T11:19:32.673 INFO:teuthology.orchestra.run.smithi204.stdout:45 files removed 2024-09-16T11:19:32.700 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-09-16T11:19:32.701 INFO:teuthology.orchestra.run.smithi053.stdout:45 files removed 2024-09-16T11:19:32.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2024-09-16T11:19:33.264 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 284 kB/s | 77 kB 00:00 2024-09-16T11:19:33.328 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 258 kB/s | 77 kB 00:00 2024-09-16T11:19:33.678 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-09-16T11:19:33.681 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-09-16T11:19:33.987 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 6.6 kB/s | 1.8 kB 00:00 2024-09-16T11:19:33.993 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 6.5 kB/s | 1.8 kB 00:00 2024-09-16T11:19:34.158 INFO:teuthology.orchestra.run.smithi204.stdout:Copr repo for el9 owned by ceph 60 kB/s | 7.6 kB 00:00 2024-09-16T11:19:34.171 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for el9 owned by ceph 52 kB/s | 7.6 kB 00:00 2024-09-16T11:19:35.141 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.2 MB 00:00 2024-09-16T11:19:36.105 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-16T11:19:36.959 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-16T11:19:39.506 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 20 MB 00:02 2024-09-16T11:19:42.410 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T11:19:44.002 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-16T11:19:44.465 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.5 MB 00:00 2024-09-16T11:19:44.882 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T11:19:46.499 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-16T11:19:47.819 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-16T11:19:50.346 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-16T11:19:52.804 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:19:52.819 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:19:52.819 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T11:19:52.820 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T11:19:52.821 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout:Install 22 Packages 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-09-16T11:19:52.822 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:19:52.823 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 58 M 2024-09-16T11:19:52.823 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:19:53.459 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T11:19:53.589 INFO:teuthology.orchestra.run.smithi204.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-16T11:19:53.647 INFO:teuthology.orchestra.run.smithi204.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 432 kB/s | 25 kB 00:00 2024-09-16T11:19:53.747 INFO:teuthology.orchestra.run.smithi204.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.2 MB/s | 637 kB 00:00 2024-09-16T11:19:53.906 INFO:teuthology.orchestra.run.smithi204.stdout:(4/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 18 MB/s | 14 MB 00:00 2024-09-16T11:19:53.931 INFO:teuthology.orchestra.run.smithi204.stdout:(5/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.4 MB/s | 458 kB 00:00 2024-09-16T11:19:54.007 INFO:teuthology.orchestra.run.smithi204.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 602 kB/s | 45 kB 00:00 2024-09-16T11:19:54.099 INFO:teuthology.orchestra.run.smithi204.stdout:(7/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 26 MB/s | 4.9 MB 00:00 2024-09-16T11:19:54.124 INFO:teuthology.orchestra.run.smithi204.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 1.0 MB/s | 120 kB 00:00 2024-09-16T11:19:54.149 INFO:teuthology.orchestra.run.smithi204.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 3.0 MB/s | 156 kB 00:00 2024-09-16T11:19:54.183 INFO:teuthology.orchestra.run.smithi204.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 5.3 MB/s | 320 kB 00:00 2024-09-16T11:19:54.208 INFO:teuthology.orchestra.run.smithi204.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 4.9 MB/s | 295 kB 00:00 2024-09-16T11:19:54.234 INFO:teuthology.orchestra.run.smithi204.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.9 MB/s | 100 kB 00:00 2024-09-16T11:19:54.551 INFO:teuthology.orchestra.run.smithi204.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 16 MB/s | 22 MB 00:01 2024-09-16T11:19:54.576 INFO:teuthology.orchestra.run.smithi204.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-09-16T11:19:54.634 INFO:teuthology.orchestra.run.smithi204.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 376 kB/s | 160 kB 00:00 2024-09-16T11:19:54.685 INFO:teuthology.orchestra.run.smithi204.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-16T11:19:54.718 INFO:teuthology.orchestra.run.smithi204.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-16T11:19:54.760 INFO:teuthology.orchestra.run.smithi204.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-16T11:19:54.785 INFO:teuthology.orchestra.run.smithi204.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-09-16T11:19:54.819 INFO:teuthology.orchestra.run.smithi204.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-09-16T11:19:54.844 INFO:teuthology.orchestra.run.smithi204.stdout:(21/24): liboath-2.6.7-2.el9.x86_64.rpm 392 kB/s | 49 kB 00:00 2024-09-16T11:19:54.894 INFO:teuthology.orchestra.run.smithi204.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-16T11:19:55.045 INFO:teuthology.orchestra.run.smithi204.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-09-16T11:19:55.115 INFO:teuthology.orchestra.run.smithi204.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-16T11:19:55.121 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:19:55.122 INFO:teuthology.orchestra.run.smithi204.stdout:Total 25 MB/s | 58 MB 00:02 2024-09-16T11:19:55.186 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:19:55.220 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:19:55.220 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:19:55.871 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:19:55.872 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:19:56.020 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:19:56.037 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:19:56.037 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:19:56.037 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T11:19:56.038 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-16T11:19:56.039 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout:Install 22 Packages 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:19:56.040 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 58 M 2024-09-16T11:19:56.041 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:19:57.113 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:19:57.511 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T11:19:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:19:57.664 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:19:57.742 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:19:57.790 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:19:57.878 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:19:57.924 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T11:19:57.962 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T11:19:58.302 INFO:teuthology.orchestra.run.smithi053.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 2.8 MB/s | 5.0 MB 00:01 2024-09-16T11:19:58.385 INFO:teuthology.orchestra.run.smithi053.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 303 kB/s | 25 kB 00:00 2024-09-16T11:19:58.398 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T11:19:58.427 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:19:58.514 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:19:58.551 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T11:19:58.560 INFO:teuthology.orchestra.run.smithi053.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 3.6 MB/s | 637 kB 00:00 2024-09-16T11:19:58.594 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T11:19:58.632 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:19:58.693 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:19:58.721 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:19:58.728 INFO:teuthology.orchestra.run.smithi053.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.7 MB/s | 458 kB 00:00 2024-09-16T11:19:58.787 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:19:58.826 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T11:19:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T11:19:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T11:19:59.189 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T11:19:59.226 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:19:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:19:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T11:19:59.419 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T11:19:59.686 INFO:teuthology.orchestra.run.smithi053.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 5.1 MB/s | 4.9 MB 00:00 2024-09-16T11:19:59.745 INFO:teuthology.orchestra.run.smithi053.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 777 kB/s | 45 kB 00:00 2024-09-16T11:19:59.803 INFO:teuthology.orchestra.run.smithi053.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 2.0 MB/s | 120 kB 00:00 2024-09-16T11:19:59.870 INFO:teuthology.orchestra.run.smithi053.stdout:(8/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.3 MB/s | 156 kB 00:00 2024-09-16T11:19:59.962 INFO:teuthology.orchestra.run.smithi053.stdout:(9/24): python3-rados-17.2.7-1643.gd75f2382.el9 3.4 MB/s | 320 kB 00:00 2024-09-16T11:20:00.112 INFO:teuthology.orchestra.run.smithi053.stdout:(10/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9 4.0 MB/s | 14 MB 00:03 2024-09-16T11:20:00.138 INFO:teuthology.orchestra.run.smithi053.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 1.6 MB/s | 295 kB 00:00 2024-09-16T11:20:00.371 INFO:teuthology.orchestra.run.smithi053.stdout:(12/24): ceph-common-17.2.7-1643.gd75f2382.el9. 5.9 MB/s | 22 MB 00:03 2024-09-16T11:20:00.397 INFO:teuthology.orchestra.run.smithi053.stdout:(13/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 351 kB/s | 100 kB 00:00 2024-09-16T11:20:00.530 INFO:teuthology.orchestra.run.smithi053.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 286 kB/s | 45 kB 00:00 2024-09-16T11:20:00.563 INFO:teuthology.orchestra.run.smithi053.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 376 kB/s | 160 kB 00:00 2024-09-16T11:20:00.614 INFO:teuthology.orchestra.run.smithi053.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-16T11:20:00.656 INFO:teuthology.orchestra.run.smithi053.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-16T11:20:00.698 INFO:teuthology.orchestra.run.smithi053.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-16T11:20:00.748 INFO:teuthology.orchestra.run.smithi053.stdout:(19/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-16T11:20:00.773 INFO:teuthology.orchestra.run.smithi053.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 422 kB/s | 49 kB 00:00 2024-09-16T11:20:00.798 INFO:teuthology.orchestra.run.smithi053.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 729 kB/s | 67 kB 00:00 2024-09-16T11:20:00.865 INFO:teuthology.orchestra.run.smithi053.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-16T11:20:00.982 INFO:teuthology.orchestra.run.smithi053.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 16 MB/s | 2.9 MB 00:00 2024-09-16T11:20:01.099 INFO:teuthology.orchestra.run.smithi053.stdout:(24/24): librados2-17.2.7-1643.gd75f2382.el9.x8 9.5 MB/s | 3.1 MB 00:00 2024-09-16T11:20:01.105 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:01.105 INFO:teuthology.orchestra.run.smithi053.stdout:Total 11 MB/s | 58 MB 00:05 2024-09-16T11:20:01.171 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:20:01.209 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:20:01.209 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:20:01.862 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:20:01.874 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:20:01.875 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:20:01.925 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:20:02.452 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:20:02.495 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:20:02.589 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:20:02.589 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T11:20:02.589 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T11:20:02.589 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:02.621 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:20:03.135 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:20:03.482 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-16T11:20:03.585 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:20:03.699 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:20:03.786 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:20:03.811 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:20:03.901 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:20:03.947 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-16T11:20:03.983 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-16T11:20:04.447 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-16T11:20:04.481 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:20:04.553 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-16T11:20:04.589 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-16T11:20:04.633 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-16T11:20:04.670 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:20:04.742 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:20:04.767 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:20:04.858 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-16T11:20:04.904 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-16T11:20:04.943 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-16T11:20:04.984 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-16T11:20:05.293 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-16T11:20:05.338 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:20:05.429 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-16T11:20:05.505 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-16T11:20:05.532 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-16T11:20:08.087 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:20:08.128 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:20:08.693 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-16T11:20:08.731 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:20:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-16T11:20:08.829 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T11:20:08.829 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T11:20:08.829 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:08.865 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:20:12.904 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:20:12.904 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-09-16T11:20:12.904 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-09-16T11:20:12.904 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-09-16T11:20:12.904 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-09-16T11:20:12.904 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-09-16T11:20:12.905 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-09-16T11:20:12.905 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-09-16T11:20:12.905 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:14.234 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:20:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:20:14.304 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:14.304 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T11:20:14.304 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:20:14.305 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:20:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:20:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:20:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T11:20:15.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T11:20:15.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T11:20:15.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:20:15.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T11:20:15.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T11:20:16.433 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T11:20:16.434 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.435 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.436 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:16.436 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T11:20:16.436 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:16.436 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:16.799 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-09-16T11:20:17.367 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:27 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:17.508 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T11:20:17.509 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 48 M 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 193 M 2024-09-16T11:20:17.510 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:20:17.933 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-16T11:20:18.026 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 690 kB/s | 163 kB 00:00 2024-09-16T11:20:18.051 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 544 kB/s | 64 kB 00:00 2024-09-16T11:20:19.035 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:20:19.035 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-09-16T11:20:19.035 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-09-16T11:20:19.036 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-09-16T11:20:19.036 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-09-16T11:20:19.036 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-09-16T11:20:19.036 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-09-16T11:20:19.036 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-09-16T11:20:19.036 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:20.401 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 18 MB/s | 48 MB 00:02 2024-09-16T11:20:20.403 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:20.404 INFO:teuthology.orchestra.run.smithi204.stdout:Total 17 MB/s | 48 MB 00:02 2024-09-16T11:20:20.406 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:20:20.432 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:20:20.451 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:20:20.451 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:20:20.477 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-16T11:20:20.478 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:20.478 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T11:20:20.478 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:20:20.478 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T11:20:20.478 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:20.479 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:20:20.576 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-16T11:20:20.576 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:20:20.816 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:20:20.817 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:20:21.349 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:20:21.436 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T11:20:21.475 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T11:20:21.503 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:20:21.901 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-16T11:20:21.902 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-16T11:20:21.903 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-16T11:20:21.904 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-16T11:20:22.766 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-16T11:20:22.766 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:22.766 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-09-16T11:20:22.766 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.766 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.766 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.767 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:22.768 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:23.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2024-09-16T11:20:23.705 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:23.842 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:23.842 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:23.842 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-09-16T11:20:23.843 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:23.844 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 48 M 2024-09-16T11:20:23.844 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 193 M 2024-09-16T11:20:23.844 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:20:24.264 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-16T11:20:24.306 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T11:20:24.673 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 284 kB/s | 163 kB 00:00 2024-09-16T11:20:26.131 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:20:26.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:20:28.073 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 48 MB 00:03 2024-09-16T11:20:28.075 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:28.075 INFO:teuthology.orchestra.run.smithi053.stdout:Total 11 MB/s | 48 MB 00:04 2024-09-16T11:20:28.104 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:20:28.125 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:20:28.125 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:20:28.381 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:20:28.382 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T11:20:28.382 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T11:20:28.382 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T11:20:28.509 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:20:28.509 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:28.701 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:28.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-09-16T11:20:29.069 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:20:29.142 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T11:20:29.181 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T11:20:29.207 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:20:29.574 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:29.718 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:29.720 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:29.720 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:20:29.720 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T11:20:29.721 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T11:20:29.722 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T11:20:29.723 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:Install 34 Packages 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 27 M 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 96 M 2024-09-16T11:20:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:20:30.435 INFO:teuthology.orchestra.run.smithi204.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 50 kB/s | 6.5 kB 00:00 2024-09-16T11:20:30.619 INFO:teuthology.orchestra.run.smithi204.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.5 MB/s | 1.4 MB 00:00 2024-09-16T11:20:30.669 INFO:teuthology.orchestra.run.smithi204.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-16T11:20:30.761 INFO:teuthology.orchestra.run.smithi204.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 2.6 MB/s | 240 kB 00:00 2024-09-16T11:20:30.828 INFO:teuthology.orchestra.run.smithi204.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 10 MB/s | 3.9 MB 00:00 2024-09-16T11:20:31.128 INFO:teuthology.orchestra.run.smithi204.stdout:(6/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-16T11:20:31.445 INFO:teuthology.orchestra.run.smithi204.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 370 kB/s | 253 kB 00:00 2024-09-16T11:20:31.562 INFO:teuthology.orchestra.run.smithi204.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-16T11:20:31.587 INFO:teuthology.orchestra.run.smithi204.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-16T11:20:31.687 INFO:teuthology.orchestra.run.smithi204.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 996 kB/s | 125 kB 00:00 2024-09-16T11:20:31.788 INFO:teuthology.orchestra.run.smithi204.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-16T11:20:31.880 INFO:teuthology.orchestra.run.smithi204.stdout:(12/34): python3-mako-1.1.4-6.el9.noarch.rpm 898 kB/s | 172 kB 00:00 2024-09-16T11:20:31.897 INFO:teuthology.orchestra.run.smithi204.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-16T11:20:31.938 INFO:teuthology.orchestra.run.smithi204.stdout:(14/34): python3-cryptography-36.0.1-4.el9.x86_ 1.1 MB/s | 1.2 MB 00:01 2024-09-16T11:20:31.964 INFO:teuthology.orchestra.run.smithi204.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 920 kB/s | 77 kB 00:00 2024-09-16T11:20:31.981 INFO:teuthology.orchestra.run.smithi204.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 608 kB/s | 51 kB 00:00 2024-09-16T11:20:32.014 INFO:teuthology.orchestra.run.smithi204.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 874 kB/s | 43 kB 00:00 2024-09-16T11:20:32.048 INFO:teuthology.orchestra.run.smithi204.stdout:(18/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-16T11:20:32.073 INFO:teuthology.orchestra.run.smithi204.stdout:(19/34): python3-toml-0.10.2-6.el9.noarch.rpm 310 kB/s | 42 kB 00:00 2024-09-16T11:20:32.098 INFO:teuthology.orchestra.run.smithi204.stdout:(20/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-16T11:20:32.133 INFO:teuthology.orchestra.run.smithi204.stdout:(21/34): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-16T11:20:32.157 INFO:teuthology.orchestra.run.smithi204.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 452 kB/s | 23 kB 00:00 2024-09-16T11:20:32.183 INFO:teuthology.orchestra.run.smithi204.stdout:(23/34): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-16T11:20:32.208 INFO:teuthology.orchestra.run.smithi204.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 263 kB/s | 19 kB 00:00 2024-09-16T11:20:32.233 INFO:teuthology.orchestra.run.smithi204.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-16T11:20:32.258 INFO:teuthology.orchestra.run.smithi204.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-09-16T11:20:32.284 INFO:teuthology.orchestra.run.smithi204.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T11:20:32.309 INFO:teuthology.orchestra.run.smithi204.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T11:20:32.335 INFO:teuthology.orchestra.run.smithi204.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T11:20:32.360 INFO:teuthology.orchestra.run.smithi204.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T11:20:32.385 INFO:teuthology.orchestra.run.smithi204.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T11:20:32.410 INFO:teuthology.orchestra.run.smithi204.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-16T11:20:32.435 INFO:teuthology.orchestra.run.smithi204.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T11:20:32.469 INFO:teuthology.orchestra.run.smithi204.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-16T11:20:32.474 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:32.474 INFO:teuthology.orchestra.run.smithi204.stdout:Total 9.9 MB/s | 27 MB 00:02 2024-09-16T11:20:32.608 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:20:32.646 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:20:32.647 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:20:33.056 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:20:33.057 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:20:33.885 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-16T11:20:33.959 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:20:34.269 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:20:34.330 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T11:20:34.363 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T11:20:34.404 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T11:20:34.496 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T11:20:34.599 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T11:20:34.652 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T11:20:34.687 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T11:20:34.717 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T11:20:34.751 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T11:20:34.785 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T11:20:34.825 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T11:20:34.860 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T11:20:34.898 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T11:20:34.952 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T11:20:34.983 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:20:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T11:20:35.212 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T11:20:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T11:20:35.353 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T11:20:35.392 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T11:20:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T11:20:35.496 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T11:20:35.684 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T11:20:35.730 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T11:20:35.808 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T11:20:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T11:20:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T11:20:36.086 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T11:20:36.236 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T11:20:36.267 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:20:36.294 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:20:36.294 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:36.294 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T11:20:36.294 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:20:36.294 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:20:36.294 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:36.295 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-16T11:20:36.295 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-16T11:20:36.295 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-16T11:20:36.295 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T11:20:36.653 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T11:20:36.654 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:36.654 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:36.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2024-09-16T11:20:37.491 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:44 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:37.610 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:20:37.631 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-16T11:20:37.634 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T11:20:37.635 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T11:20:37.636 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:37.637 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:20:37.638 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:37.638 INFO:teuthology.orchestra.run.smithi053.stdout:Install 34 Packages 2024-09-16T11:20:37.638 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:37.638 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:20:37.638 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 27 M 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 96 M 2024-09-16T11:20:37.639 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:20:38.011 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:20:38.035 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:20:38.035 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:38.035 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T11:20:38.035 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:20:38.036 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:20:38.036 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:38.065 INFO:teuthology.orchestra.run.smithi053.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2024-09-16T11:20:38.236 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:20:38.260 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:20:38.261 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:38.261 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T11:20:38.261 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:20:38.261 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:20:38.261 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:38.325 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:20:38.624 INFO:teuthology.orchestra.run.smithi053.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 1.9 MB/s | 1.4 MB 00:00 2024-09-16T11:20:38.757 INFO:teuthology.orchestra.run.smithi053.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2024-09-16T11:20:38.832 INFO:teuthology.orchestra.run.smithi053.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 3.1 MB/s | 240 kB 00:00 2024-09-16T11:20:38.950 INFO:teuthology.orchestra.run.smithi053.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-16T11:20:39.033 INFO:teuthology.orchestra.run.smithi053.stdout:(6/34): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-16T11:20:39.075 INFO:teuthology.orchestra.run.smithi053.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T11:20:39.117 INFO:teuthology.orchestra.run.smithi053.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T11:20:39.150 INFO:teuthology.orchestra.run.smithi053.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 3.7 MB/s | 125 kB 00:00 2024-09-16T11:20:39.192 INFO:teuthology.orchestra.run.smithi053.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 5.0 MB/s | 215 kB 00:00 2024-09-16T11:20:39.318 INFO:teuthology.orchestra.run.smithi053.stdout:(11/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T11:20:39.360 INFO:teuthology.orchestra.run.smithi053.stdout:(12/34): python3-markupsafe-1.1.1-12.el9.x86_64 833 kB/s | 35 kB 00:00 2024-09-16T11:20:39.393 INFO:teuthology.orchestra.run.smithi053.stdout:(13/34): python3-packaging-20.9-5.el9.noarch.rp 2.3 MB/s | 77 kB 00:00 2024-09-16T11:20:39.435 INFO:teuthology.orchestra.run.smithi053.stdout:(14/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-16T11:20:39.494 INFO:teuthology.orchestra.run.smithi053.stdout:(15/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86 2.7 MB/s | 3.9 MB 00:01 2024-09-16T11:20:39.519 INFO:teuthology.orchestra.run.smithi053.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 499 kB/s | 42 kB 00:00 2024-09-16T11:20:39.561 INFO:teuthology.orchestra.run.smithi053.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 650 kB/s | 43 kB 00:00 2024-09-16T11:20:39.595 INFO:teuthology.orchestra.run.smithi053.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-16T11:20:39.620 INFO:teuthology.orchestra.run.smithi053.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-09-16T11:20:39.637 INFO:teuthology.orchestra.run.smithi053.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-16T11:20:39.662 INFO:teuthology.orchestra.run.smithi053.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-09-16T11:20:39.687 INFO:teuthology.orchestra.run.smithi053.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 925 kB/s | 23 kB 00:00 2024-09-16T11:20:39.712 INFO:teuthology.orchestra.run.smithi053.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T11:20:39.737 INFO:teuthology.orchestra.run.smithi053.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T11:20:39.763 INFO:teuthology.orchestra.run.smithi053.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-16T11:20:39.780 INFO:teuthology.orchestra.run.smithi053.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-16T11:20:39.808 INFO:teuthology.orchestra.run.smithi053.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 586 kB/s | 16 kB 00:00 2024-09-16T11:20:39.834 INFO:teuthology.orchestra.run.smithi053.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-16T11:20:39.859 INFO:teuthology.orchestra.run.smithi053.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T11:20:39.885 INFO:teuthology.orchestra.run.smithi053.stdout:(30/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.2 MB/s | 272 kB 00:00 2024-09-16T11:20:39.910 INFO:teuthology.orchestra.run.smithi053.stdout:(31/34): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T11:20:39.935 INFO:teuthology.orchestra.run.smithi053.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-16T11:20:39.960 INFO:teuthology.orchestra.run.smithi053.stdout:(33/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-09-16T11:20:40.034 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T11:20:40.035 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T11:20:40.036 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T11:20:40.037 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T11:20:40.038 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T11:20:40.486 INFO:teuthology.orchestra.run.smithi053.stdout:(34/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 8.2 MB/s | 15 MB 00:01 2024-09-16T11:20:40.490 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:40.491 INFO:teuthology.orchestra.run.smithi053.stdout:Total 9.5 MB/s | 27 MB 00:02 2024-09-16T11:20:40.627 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:20:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:20:40.669 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:20:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T11:20:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:41.046 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:20:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T11:20:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T11:20:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:41.049 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:41.082 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:20:41.082 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:20:41.413 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-09-16T11:20:41.986 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:42.077 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:20:42.119 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:42.120 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:20:42.120 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:42.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-09-16T11:20:42.342 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:20:42.414 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-16T11:20:42.446 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-16T11:20:42.479 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-16T11:20:42.574 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-16T11:20:42.652 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-16T11:20:42.707 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-16T11:20:42.742 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-16T11:20:42.746 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:42.771 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-16T11:20:42.797 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-16T11:20:42.831 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-16T11:20:42.871 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-16T11:20:42.878 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:42.879 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:42.879 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:20:42.879 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:42.879 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:20:42.879 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 83 k 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 375 k 2024-09-16T11:20:42.880 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:20:42.915 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-16T11:20:42.949 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-16T11:20:42.998 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-16T11:20:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:20:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-16T11:20:43.255 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 220 kB/s | 83 kB 00:00 2024-09-16T11:20:43.256 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:43.256 INFO:teuthology.orchestra.run.smithi204.stdout:Total 219 kB/s | 83 kB 00:00 2024-09-16T11:20:43.256 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:20:43.265 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:20:43.265 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:20:43.265 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:20:43.265 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:20:43.273 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-16T11:20:43.343 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:20:43.356 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-16T11:20:43.416 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-16T11:20:43.454 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-16T11:20:43.513 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-16T11:20:43.576 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-16T11:20:43.697 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:43.779 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-16T11:20:43.791 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:43.840 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-16T11:20:43.921 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-16T11:20:44.010 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-16T11:20:44.047 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-16T11:20:44.145 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:44.168 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-16T11:20:44.333 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-16T11:20:44.382 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:44.382 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:44.382 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:20:44.382 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:44.382 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:44.382 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:44.383 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:20:44.409 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-16T11:20:44.409 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:44.409 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T11:20:44.409 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:20:44.410 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T11:20:44.410 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:44.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T11:20:45.161 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:45.297 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:45.298 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:20:45.299 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:45.299 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:20:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:45.299 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 137 k 2024-09-16T11:20:45.299 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 379 k 2024-09-16T11:20:45.299 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:20:45.745 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 308 kB/s | 137 kB 00:00 2024-09-16T11:20:45.746 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:45.746 INFO:teuthology.orchestra.run.smithi204.stdout:Total 307 kB/s | 137 kB 00:00 2024-09-16T11:20:45.746 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:20:45.754 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:20:45.754 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:20:45.808 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:20:45.810 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:20:45.811 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:20:45.836 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-16T11:20:45.836 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:45.837 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T11:20:45.837 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:20:45.837 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T11:20:45.837 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:45.955 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:20:46.021 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T11:20:46.046 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:46.234 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:20:46.259 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-16T11:20:46.260 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:46.260 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T11:20:46.260 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:20:46.260 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T11:20:46.260 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:46.467 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:20:46.493 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-16T11:20:46.493 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:46.493 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T11:20:46.493 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:20:46.493 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T11:20:46.493 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:46.560 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:20:46.979 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:20:46.979 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:46.980 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:20:46.980 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:46.980 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:46.980 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:47.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-09-16T11:20:47.765 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:47.859 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:20:47.900 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:47.901 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:20:47.901 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:47.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T11:20:48.301 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-16T11:20:48.301 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-16T11:20:48.301 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-16T11:20:48.301 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-16T11:20:48.301 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-16T11:20:48.302 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-16T11:20:48.303 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-16T11:20:48.304 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-16T11:20:48.530 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:48.665 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T11:20:48.666 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Install 7 Packages 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.7 M 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 80 M 2024-09-16T11:20:48.667 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:20:49.237 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-16T11:20:49.237 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:49.237 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:20:49.237 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T11:20:49.238 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T11:20:49.239 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:49.240 INFO:teuthology.orchestra.run.smithi204.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 60 kB/s | 24 kB 00:00 2024-09-16T11:20:49.256 INFO:teuthology.orchestra.run.smithi204.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 35 kB/s | 15 kB 00:00 2024-09-16T11:20:49.299 INFO:teuthology.orchestra.run.smithi204.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-09-16T11:20:49.315 INFO:teuthology.orchestra.run.smithi204.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 699 kB/s | 41 kB 00:00 2024-09-16T11:20:49.332 INFO:teuthology.orchestra.run.smithi204.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 918 kB/s | 31 kB 00:00 2024-09-16T11:20:49.358 INFO:teuthology.orchestra.run.smithi204.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-16T11:20:49.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2024-09-16T11:20:50.125 INFO:teuthology.orchestra.run.smithi204.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 2.6 MB/s | 3.4 MB 00:01 2024-09-16T11:20:50.129 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:50.129 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.5 MB/s | 3.7 MB 00:01 2024-09-16T11:20:50.130 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:20:50.139 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:20:50.139 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:20:50.193 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:50.229 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:20:50.230 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:20:50.286 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:20:50.329 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:50.330 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:20:50.330 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:50.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2024-09-16T11:20:50.533 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:20:50.584 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T11:20:50.618 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T11:20:50.665 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T11:20:50.697 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T11:20:50.725 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T11:20:50.984 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:51.121 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:51.122 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:20:51.122 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:51.122 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:20:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-16T11:20:51.122 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 83 k 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 375 k 2024-09-16T11:20:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:20:51.324 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 410 kB/s | 83 kB 00:00 2024-09-16T11:20:51.325 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:51.325 INFO:teuthology.orchestra.run.smithi053.stdout:Total 407 kB/s | 83 kB 00:00 2024-09-16T11:20:51.325 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:20:51.331 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:20:51.331 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:20:51.333 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:20:51.333 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:20:51.415 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:20:51.724 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T11:20:51.763 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:20:51.885 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:52.392 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:52.651 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:20:52.652 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:52.652 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:20:52.652 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:52.652 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:52.652 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T11:20:52.782 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T11:20:52.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T11:20:53.055 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T11:20:53.055 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:53.056 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:20:53.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T11:20:53.475 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:53.613 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 137 k 2024-09-16T11:20:53.614 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 379 k 2024-09-16T11:20:53.615 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:20:53.807 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:03 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:20:53.940 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:20:53.941 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-16T11:20:53.941 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T11:20:53.942 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-09-16T11:20:53.943 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:20:53.944 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 38 M 2024-09-16T11:20:53.944 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 212 M 2024-09-16T11:20:53.944 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:20:54.124 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 270 kB/s | 137 kB 00:00 2024-09-16T11:20:54.124 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:54.125 INFO:teuthology.orchestra.run.smithi053.stdout:Total 269 kB/s | 137 kB 00:00 2024-09-16T11:20:54.125 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:20:54.134 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:20:54.134 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:20:54.195 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:20:54.196 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:20:54.350 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:20:54.433 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:20:54.460 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:20:54.460 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:20:54.460 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T11:20:54.460 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:54.833 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 257 kB/s | 191 kB 00:00 2024-09-16T11:20:54.966 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 912 kB/s | 801 kB 00:00 2024-09-16T11:20:55.116 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-09-16T11:20:55.225 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-09-16T11:20:55.334 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 355 kB/s | 38 kB 00:00 2024-09-16T11:20:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-16T11:20:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:55.435 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:20:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:20:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:55.436 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:55.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2024-09-16T11:20:55.692 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 4.6 MB/s | 7.4 MB 00:01 2024-09-16T11:20:56.244 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:03 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:56.276 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 5.2 MB/s | 4.9 MB 00:00 2024-09-16T11:20:56.336 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:20:56.381 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:56.382 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:20:56.382 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:20:56.401 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 345 kB/s | 245 kB 00:00 2024-09-16T11:20:56.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T11:20:56.610 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-09-16T11:20:56.793 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 1.6 MB/s | 3.0 MB 00:01 2024-09-16T11:20:57.031 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:04 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:20:57.170 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-16T11:20:57.171 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout:Install 7 Packages 2024-09-16T11:20:57.172 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:20:57.173 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.7 M 2024-09-16T11:20:57.173 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 80 M 2024-09-16T11:20:57.173 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:20:57.403 INFO:teuthology.orchestra.run.smithi053.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 143 kB/s | 24 kB 00:00 2024-09-16T11:20:57.428 INFO:teuthology.orchestra.run.smithi053.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 75 kB/s | 15 kB 00:00 2024-09-16T11:20:57.470 INFO:teuthology.orchestra.run.smithi053.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-16T11:20:57.495 INFO:teuthology.orchestra.run.smithi053.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-16T11:20:57.521 INFO:teuthology.orchestra.run.smithi053.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-16T11:20:57.546 INFO:teuthology.orchestra.run.smithi053.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T11:20:57.696 INFO:teuthology.orchestra.run.smithi053.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 7.2 MB/s | 3.4 MB 00:00 2024-09-16T11:20:57.699 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:20:57.699 INFO:teuthology.orchestra.run.smithi053.stdout:Total 6.9 MB/s | 3.7 MB 00:00 2024-09-16T11:20:57.700 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:20:57.708 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:20:57.708 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:20:57.794 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:20:57.794 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:20:58.107 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:20:58.180 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-16T11:20:58.215 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-16T11:20:58.263 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-16T11:20:58.293 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-16T11:20:58.322 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-16T11:20:59.377 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-16T11:20:59.408 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-16T11:21:00.448 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-16T11:21:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-16T11:21:00.743 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:00.744 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:00.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T11:21:01.168 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.0 MB/s | 5.1 MB 00:04 2024-09-16T11:21:01.540 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:01.681 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T11:21:01.682 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout:Install 12 Packages 2024-09-16T11:21:01.683 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:01.684 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 38 M 2024-09-16T11:21:01.684 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 212 M 2024-09-16T11:21:01.684 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:02.695 INFO:teuthology.orchestra.run.smithi053.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 254 kB/s | 191 kB 00:00 2024-09-16T11:21:02.795 INFO:teuthology.orchestra.run.smithi053.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 329 kB/s | 33 kB 00:00 2024-09-16T11:21:02.962 INFO:teuthology.orchestra.run.smithi053.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 7.3 MB/s | 7.4 MB 00:01 2024-09-16T11:21:03.004 INFO:teuthology.orchestra.run.smithi053.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 755 kB/s | 801 kB 00:01 2024-09-16T11:21:03.071 INFO:teuthology.orchestra.run.smithi053.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-09-16T11:21:03.105 INFO:teuthology.orchestra.run.smithi053.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 385 kB/s | 38 kB 00:00 2024-09-16T11:21:03.196 INFO:teuthology.orchestra.run.smithi053.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.5 MB/s | 3.0 MB 00:00 2024-09-16T11:21:03.263 INFO:teuthology.orchestra.run.smithi053.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-09-16T11:21:03.347 INFO:teuthology.orchestra.run.smithi053.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-09-16T11:21:03.372 INFO:teuthology.orchestra.run.smithi053.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-16T11:21:03.639 INFO:teuthology.orchestra.run.smithi053.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 56 MB/s | 16 MB 00:00 2024-09-16T11:21:04.088 INFO:teuthology.orchestra.run.smithi053.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.9 MB/s | 4.9 MB 00:01 2024-09-16T11:21:04.089 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:04.089 INFO:teuthology.orchestra.run.smithi053.stdout:Total 16 MB/s | 38 MB 00:02 2024-09-16T11:21:04.460 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:04.489 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:04.489 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:05.014 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:05.794 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:05.865 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T11:21:05.913 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:21:06.043 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:21:06.089 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T11:21:06.282 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T11:21:06.312 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T11:21:06.392 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T11:21:07.104 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T11:21:07.231 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T11:21:08.563 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:21:09.637 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T11:21:09.661 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:21:12.221 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:21:12.221 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T11:21:12.221 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:21:12.222 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T11:21:12.626 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T11:21:12.626 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:12.626 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:12.626 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:12.627 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:13.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2024-09-16T11:21:13.616 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:13.756 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T11:21:13.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-09-16T11:21:13.758 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:13.759 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.2 M 2024-09-16T11:21:13.759 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 26 M 2024-09-16T11:21:13.759 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:14.495 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 254 kB/s | 19 kB 00:00 2024-09-16T11:21:14.509 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 930 kB/s | 16 MB 00:17 2024-09-16T11:21:14.511 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:14.511 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 278 kB/s | 26 kB 00:00 2024-09-16T11:21:14.511 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.9 MB/s | 38 MB 00:20 2024-09-16T11:21:14.570 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 314 kB/s | 47 kB 00:00 2024-09-16T11:21:14.595 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-16T11:21:14.620 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T11:21:14.645 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T11:21:14.696 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 429 kB/s | 32 kB 00:00 2024-09-16T11:21:14.721 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-09-16T11:21:14.747 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-16T11:21:14.788 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-16T11:21:14.814 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-16T11:21:14.830 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-09-16T11:21:14.868 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:14.895 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:14.895 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:14.905 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.6 MB/s | 1.0 MB 00:00 2024-09-16T11:21:14.909 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:14.909 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-16T11:21:14.981 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:14.999 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:14.999 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:15.094 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:15.094 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:15.401 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:15.401 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:15.605 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:15.777 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T11:21:15.834 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T11:21:15.889 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T11:21:15.950 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T11:21:15.992 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T11:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T11:21:16.177 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:16.207 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T11:21:16.264 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T11:21:16.268 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T11:21:16.306 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:21:16.403 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:21:16.442 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T11:21:16.626 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T11:21:16.666 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T11:21:16.743 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T11:21:17.028 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T11:21:17.081 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T11:21:17.115 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T11:21:17.161 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T11:21:17.184 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:21:17.401 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T11:21:17.513 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T11:21:17.996 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T11:21:17.997 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T11:21:17.997 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T11:21:17.997 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T11:21:18.331 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T11:21:18.331 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:18.331 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T11:21:18.332 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T11:21:18.333 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:18.333 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:18.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T11:21:18.771 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:21:19.160 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:19.298 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T11:21:19.299 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T11:21:19.300 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T11:21:19.300 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:19.300 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:19.300 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:19.300 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-09-16T11:21:19.300 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:19.301 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 7.0 M 2024-09-16T11:21:19.301 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 33 M 2024-09-16T11:21:19.301 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:19.805 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T11:21:19.829 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T11:21:19.832 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:21:19.857 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 505 kB/s | 124 kB 00:00 2024-09-16T11:21:19.915 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 994 kB/s | 58 kB 00:00 2024-09-16T11:21:19.949 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-16T11:21:19.991 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.3 MB/s | 548 kB 00:00 2024-09-16T11:21:20.099 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-16T11:21:20.102 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:20.102 INFO:teuthology.orchestra.run.smithi053.stdout:Total 8.7 MB/s | 7.0 MB 00:00 2024-09-16T11:21:20.177 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:20.189 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:20.190 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:20.294 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:20.296 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:20.623 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:20.797 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T11:21:20.871 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T11:21:21.432 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T11:21:21.515 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T11:21:21.586 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T11:21:21.621 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:21:22.280 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-16T11:21:22.280 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T11:21:22.281 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T11:21:22.392 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:21:22.393 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T11:21:22.393 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T11:21:22.393 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T11:21:22.393 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T11:21:22.393 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T11:21:22.664 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:22.665 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T11:21:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:22.687 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:22.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2024-09-16T11:21:23.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2024-09-16T11:21:23.486 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:23.626 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:23.626 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-09-16T11:21:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:23.628 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 893 k 2024-09-16T11:21:23.628 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.6 M 2024-09-16T11:21:23.628 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:23.631 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:23.765 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:23.766 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:23.766 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:21:23.766 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:23.766 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T11:21:23.767 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:23.768 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.2 M 2024-09-16T11:21:23.769 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-09-16T11:21:23.769 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:23.884 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 577 kB/s | 80 kB 00:00 2024-09-16T11:21:24.018 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-16T11:21:24.018 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:24.018 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.2 MB/s | 893 kB 00:00 2024-09-16T11:21:24.031 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:24.054 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:24.054 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:24.139 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:24.139 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:24.330 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:24.359 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 223 kB/s | 26 kB 00:00 2024-09-16T11:21:24.384 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 19 kB 00:00 2024-09-16T11:21:24.409 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 287 kB/s | 47 kB 00:00 2024-09-16T11:21:24.488 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T11:21:24.493 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-16T11:21:24.518 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-16T11:21:24.551 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-09-16T11:21:24.569 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:21:24.585 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-16T11:21:24.602 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 290 kB/s | 14 kB 00:00 2024-09-16T11:21:24.627 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 5.3 MB/s | 220 kB 00:00 2024-09-16T11:21:24.653 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.1 MB/s | 279 kB 00:00 2024-09-16T11:21:24.670 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-16T11:21:24.711 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 9.4 MB/s | 1.0 MB 00:00 2024-09-16T11:21:24.737 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-09-16T11:21:24.739 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:24.739 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-16T11:21:24.805 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:24.820 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:24.820 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:24.908 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:24.912 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:21:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T11:21:25.421 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:25.549 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T11:21:25.549 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:25.549 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:25.549 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T11:21:25.549 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:25.549 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:25.602 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T11:21:25.655 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T11:21:25.711 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T11:21:25.781 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T11:21:25.822 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T11:21:25.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-volume 2024-09-16T11:21:25.924 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T11:21:26.035 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T11:21:26.091 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T11:21:26.423 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:26.563 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:26.564 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 270 k 2024-09-16T11:21:26.565 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.3 M 2024-09-16T11:21:26.565 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-16T11:21:26.789 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:26.798 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:26.799 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:26.810 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T11:21:26.826 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:26.826 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:26.863 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T11:21:26.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T11:21:26.944 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T11:21:26.966 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:21:27.047 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:27.116 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:21:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:21:27.133 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:21:27.133 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T11:21:27.133 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:27.696 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T11:21:27.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T11:21:27.698 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T11:21:28.033 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:28.034 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:28.181 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:21:28.181 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:28.181 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:28.181 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:28.181 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:28.182 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:28.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T11:21:28.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2024-09-16T11:21:28.819 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:28.953 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T11:21:28.954 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T11:21:28.955 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T11:21:28.955 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:28.955 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:28.955 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:28.955 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-09-16T11:21:28.955 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:28.956 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 7.0 M 2024-09-16T11:21:28.956 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 33 M 2024-09-16T11:21:28.956 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:29.076 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:29.214 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:29.215 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 126 k 2024-09-16T11:21:29.216 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 448 k 2024-09-16T11:21:29.216 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:29.281 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 645 kB/s | 124 kB 00:00 2024-09-16T11:21:29.314 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-16T11:21:29.373 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 992 kB/s | 58 kB 00:00 2024-09-16T11:21:29.398 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T11:21:29.426 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 599 kB/s | 126 kB 00:00 2024-09-16T11:21:29.427 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:29.427 INFO:teuthology.orchestra.run.smithi053.stdout:Total 595 kB/s | 126 kB 00:00 2024-09-16T11:21:29.427 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:29.432 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.5 MB/s | 548 kB 00:00 2024-09-16T11:21:29.433 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:29.433 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:29.501 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:29.501 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:29.615 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-16T11:21:29.617 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:29.617 INFO:teuthology.orchestra.run.smithi204.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-16T11:21:29.655 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:29.688 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:29.699 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:29.699 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:29.797 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:29.798 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:29.827 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:30.099 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:30.233 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:30.283 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T11:21:30.343 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T11:21:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:30.514 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:30.514 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:30.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2024-09-16T11:21:30.867 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T11:21:30.954 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T11:21:31.009 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T11:21:31.039 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:21:31.349 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:31.443 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:31.489 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:31.489 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:31.489 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:31.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2024-09-16T11:21:31.799 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-16T11:21:31.799 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-16T11:21:31.799 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T11:21:31.799 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T11:21:31.800 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T11:21:31.800 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T11:21:32.064 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T11:21:32.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:32.064 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:32.064 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:32.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T11:21:32.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T11:21:32.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T11:21:32.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T11:21:32.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T11:21:32.065 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:32.065 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:32.146 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:32.286 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:32.286 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:21:32.287 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:32.288 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 25 k 2024-09-16T11:21:32.288 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 110 k 2024-09-16T11:21:32.288 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:32.297 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-09-16T11:21:32.438 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 166 kB/s | 25 kB 00:00 2024-09-16T11:21:32.438 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:32.439 INFO:teuthology.orchestra.run.smithi053.stdout:Total 165 kB/s | 25 kB 00:00 2024-09-16T11:21:32.439 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:32.442 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:32.442 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:32.444 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:32.444 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:32.529 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:32.638 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:32.866 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:32.987 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:33.001 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:33.001 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:33.001 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:21:33.001 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-09-16T11:21:33.002 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:33.003 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 893 k 2024-09-16T11:21:33.003 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.6 M 2024-09-16T11:21:33.003 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:33.221 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:33.221 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:33.221 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:33.221 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:33.268 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 490 kB/s | 80 kB 00:00 2024-09-16T11:21:33.352 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 3.2 MB/s | 813 kB 00:00 2024-09-16T11:21:33.352 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:33.353 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.5 MB/s | 893 kB 00:00 2024-09-16T11:21:33.365 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:33.385 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:33.385 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:33.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2024-09-16T11:21:33.463 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:33.463 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:33.651 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:33.779 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T11:21:33.850 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:21:33.985 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:34.079 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:34.125 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:34.126 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:34.126 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:34.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2024-09-16T11:21:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-16T11:21:34.543 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-16T11:21:34.782 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:34.807 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T11:21:34.808 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:34.808 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:34.808 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T11:21:34.808 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:34.808 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:34.876 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:34.921 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:34.922 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:34.922 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:34.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2024-09-16T11:21:35.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-volume 2024-09-16T11:21:35.573 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:35.663 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:35.664 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:35.710 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:35.711 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:35.711 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:35.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2024-09-16T11:21:35.798 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:35.799 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:35.800 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:35.800 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:21:35.800 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:35.800 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 270 k 2024-09-16T11:21:35.800 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.3 M 2024-09-16T11:21:35.800 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:36.031 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-16T11:21:36.032 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:36.032 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-16T11:21:36.032 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:36.041 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:36.041 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:36.073 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:36.074 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:36.286 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:36.359 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:21:36.361 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:36.374 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:21:36.375 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:21:36.375 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T11:21:36.375 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:36.453 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:36.500 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:36.501 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:36.501 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:36.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2024-09-16T11:21:37.158 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:37.250 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:37.297 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:37.298 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:37.298 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:37.351 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-16T11:21:37.351 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:37.351 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:37.352 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-16T11:21:37.352 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:37.352 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:37.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2024-09-16T11:21:37.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-09-16T11:21:37.938 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:38.031 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:38.075 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:38.077 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:38.077 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:38.139 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2024-09-16T11:21:38.224 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:38.362 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:38.363 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:38.363 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T11:21:38.363 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:38.363 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 126 k 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 448 k 2024-09-16T11:21:38.364 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:38.579 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 590 kB/s | 126 kB 00:00 2024-09-16T11:21:38.579 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:38.579 INFO:teuthology.orchestra.run.smithi204.stdout:Total 585 kB/s | 126 kB 00:00 2024-09-16T11:21:38.580 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:38.585 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:38.585 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:38.647 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:38.647 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:38.730 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:38.798 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:38.869 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:38.870 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 82 k 2024-09-16T11:21:38.871 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 222 k 2024-09-16T11:21:38.871 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:38.885 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:39.066 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 420 kB/s | 82 kB 00:00 2024-09-16T11:21:39.066 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:39.067 INFO:teuthology.orchestra.run.smithi053.stdout:Total 417 kB/s | 82 kB 00:00 2024-09-16T11:21:39.067 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:39.074 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:39.074 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:39.141 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:39.141 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:39.289 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:39.315 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:39.400 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:39.583 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:39.583 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:39.583 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:39.584 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:39.584 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:39.584 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:39.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-09-16T11:21:39.805 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:40.088 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:40.088 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:40.088 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:40.089 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:40.089 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:40.089 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:40.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-09-16T11:21:40.393 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:40.487 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:40.533 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:40.534 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:40.534 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:40.592 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-09-16T11:21:40.929 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:41.071 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:41.072 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:41.072 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-09-16T11:21:41.072 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.9 M 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 12 M 2024-09-16T11:21:41.073 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:41.168 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:41.302 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:41.303 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:41.303 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T11:21:41.303 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:41.303 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:41.303 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 25 k 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 110 k 2024-09-16T11:21:41.304 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:41.396 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 9.1 MB/s | 2.9 MB 00:00 2024-09-16T11:21:41.397 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:41.397 INFO:teuthology.orchestra.run.smithi053.stdout:Total 9.1 MB/s | 2.9 MB 00:00 2024-09-16T11:21:41.397 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:41.406 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:41.406 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:41.467 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 153 kB/s | 25 kB 00:00 2024-09-16T11:21:41.468 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:41.468 INFO:teuthology.orchestra.run.smithi204.stdout:Total 152 kB/s | 25 kB 00:00 2024-09-16T11:21:41.468 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:41.471 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:41.471 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:41.473 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:41.473 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:41.490 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:41.490 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:41.564 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:41.664 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:41.927 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:42.020 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:42.080 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:42.107 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:42.107 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:21:42.107 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T11:21:42.107 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:21:42.107 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:21:42.108 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:42.238 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:42.238 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:42.238 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:42.238 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:42.238 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:42.239 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:42.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-09-16T11:21:42.975 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:43.065 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:43.108 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:43.109 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:43.109 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:43.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-09-16T11:21:43.172 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:43.172 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:43.172 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:43.172 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:43.172 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:43.173 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:43.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2024-09-16T11:21:43.731 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:43.820 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:43.863 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:43.864 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:43.864 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:43.924 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-09-16T11:21:44.050 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-09-16T11:21:44.191 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:44.192 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 165 k 2024-09-16T11:21:44.192 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 477 k 2024-09-16T11:21:44.192 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-09-16T11:21:44.399 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 800 kB/s | 165 kB 00:00 2024-09-16T11:21:44.400 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:44.400 INFO:teuthology.orchestra.run.smithi053.stdout:Total 792 kB/s | 165 kB 00:00 2024-09-16T11:21:44.400 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-09-16T11:21:44.408 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-09-16T11:21:44.409 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-09-16T11:21:44.476 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-09-16T11:21:44.476 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-09-16T11:21:44.486 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:54 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:44.577 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:44.620 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:44.621 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:44.621 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:44.631 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-09-16T11:21:44.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-09-16T11:21:44.741 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:45.132 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:45.246 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:55 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:45.336 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:45.379 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:45.380 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:45.381 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:45.405 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:45.405 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:45.405 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-09-16T11:21:45.405 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:45.405 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:45.405 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:45.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-09-16T11:21:45.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-09-16T11:21:46.020 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:46.109 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:46.152 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:46.153 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:46.153 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:46.211 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 11:19:53 AM UTC. 2024-09-16T11:21:46.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-09-16T11:21:46.304 INFO:teuthology.orchestra.run.smithi053.stdout:Package rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:46.349 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-09-16T11:21:46.350 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-09-16T11:21:46.350 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-09-16T11:21:46.417 DEBUG:teuthology.parallel:result is None 2024-09-16T11:21:46.779 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:46.869 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:46.913 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:46.914 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:46.914 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:46.986 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-09-16T11:21:47.553 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:57 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:47.687 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:47.687 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:47.687 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 82 k 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 222 k 2024-09-16T11:21:47.688 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:47.887 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 412 kB/s | 82 kB 00:00 2024-09-16T11:21:47.887 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:47.888 INFO:teuthology.orchestra.run.smithi204.stdout:Total 409 kB/s | 82 kB 00:00 2024-09-16T11:21:47.888 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:47.894 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:47.894 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:47.957 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:47.957 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:48.102 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:48.200 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:48.582 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:48.834 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:48.834 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:48.834 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:48.834 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:48.835 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:48.835 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:49.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-09-16T11:21:49.653 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:59 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:49.793 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:49.794 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:49.795 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:49.795 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:21:49.795 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:49.795 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.9 M 2024-09-16T11:21:49.795 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 12 M 2024-09-16T11:21:49.795 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:50.133 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-09-16T11:21:50.134 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:50.134 INFO:teuthology.orchestra.run.smithi204.stdout:Total 8.7 MB/s | 2.9 MB 00:00 2024-09-16T11:21:50.134 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:50.143 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:50.143 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:50.222 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:50.222 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:50.644 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:50.700 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:50.725 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:50.725 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T11:21:50.725 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T11:21:50.725 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:21:50.725 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T11:21:50.725 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:51.775 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:51.776 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:51.776 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:51.776 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:51.776 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:51.776 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:51.996 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-09-16T11:21:52.578 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:02 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:52.711 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-16T11:21:52.712 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-16T11:21:52.713 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-16T11:21:52.713 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:52.713 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 165 k 2024-09-16T11:21:52.713 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 477 k 2024-09-16T11:21:52.713 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-16T11:21:52.920 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 797 kB/s | 165 kB 00:00 2024-09-16T11:21:52.921 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-16T11:21:52.921 INFO:teuthology.orchestra.run.smithi204.stdout:Total 791 kB/s | 165 kB 00:00 2024-09-16T11:21:52.921 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-16T11:21:52.929 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-16T11:21:52.929 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-16T11:21:52.991 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-16T11:21:52.992 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-16T11:21:53.150 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-16T11:21:53.242 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:53.665 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:53.904 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-16T11:21:53.904 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:53.904 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-16T11:21:53.904 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-16T11:21:53.904 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:53.905 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:54.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-09-16T11:21:54.703 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:02:04 ago on Mon 16 Sep 2024 11:19:50 AM UTC. 2024-09-16T11:21:54.794 INFO:teuthology.orchestra.run.smithi204.stdout:Package rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-16T11:21:54.839 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-16T11:21:54.840 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-16T11:21:54.905 DEBUG:teuthology.parallel:result is None 2024-09-16T11:21:54.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:21:55.069 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T11:21:55.095 INFO:teuthology.orchestra.run.smithi053.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T11:21:55.096 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T11:21:55.096 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T11:21:55.098 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:21:55.248 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T11:21:55.270 INFO:teuthology.orchestra.run.smithi204.stdout:17.2.7-1643.gd75f2382.el9 2024-09-16T11:21:55.270 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-16T11:21:55.271 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-16T11:21:55.272 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T11:21:55.273 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:55.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T11:21:55.304 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:21:55.304 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T11:21:55.336 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T11:21:55.337 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:55.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T11:21:55.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T11:21:55.443 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:21:55.443 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T11:21:55.468 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T11:21:55.534 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T11:21:55.534 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:55.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T11:21:55.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T11:21:55.631 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:21:55.632 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T11:21:55.656 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T11:21:55.722 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T11:21:55.722 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:55.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T11:21:55.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T11:21:55.818 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:21:55.818 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T11:21:55.843 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T11:21:55.907 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T11:21:55.982 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T11:21:55.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-09-16T11:21:55.985 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-09-16T11:21:56.012 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T11:21:56.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T11:21:56.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T11:21:56.080 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T11:21:56.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T11:21:56.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T11:21:56.161 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-16T11:21:56.161 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T11:21:56.161 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907185', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOe7lMxi92VYCpCtte6dzjkYKO5OGVDJ7IZhNUiqKzRWoeCIfgcJG4poQUlmzBExppekzihp9JzY978AaCEVF+Q=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMCaz0LRwVAGDTbGa15b9GXxNEYnUK+l3jpCRHQyjqf7CNnXJKrdfB7vxJ/LVkYxdAuKcPrxoCK843457LUB6I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_journal.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T11:21:56.162 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T11:21:56.191 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T11:21:56.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:21:56.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:21:56.273 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:56.273 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T11:21:56.331 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T11:21:56.331 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-16 11:21:42.368621618 +0000 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-16 11:19:10.545992777 +0000 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-16 11:19:10.545992777 +0000 2024-09-16T11:21:56.387 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-16 11:19:10.545992777 +0000 2024-09-16T11:21:56.388 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T11:21:56.452 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-16T11:21:56.452 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-16T11:21:56.452 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.977e-05 s, 5.1 MB/s 2024-09-16T11:21:56.453 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T11:21:56.511 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-09-16T11:21:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-16 11:21:42.368621618 +0000 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-16 11:19:10.868986095 +0000 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-16 11:19:10.868986095 +0000 2024-09-16T11:21:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-16 11:19:10.868986095 +0000 2024-09-16T11:21:56.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T11:21:56.631 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-16T11:21:56.631 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-16T11:21:56.631 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000101832 s, 5.0 MB/s 2024-09-16T11:21:56.632 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T11:21:56.689 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-09-16T11:21:56.745 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T11:21:56.745 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-16 11:21:42.369621597 +0000 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-16 11:19:11.201979206 +0000 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-16 11:19:11.201979206 +0000 2024-09-16T11:21:56.746 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-16 11:19:11.201979206 +0000 2024-09-16T11:21:56.746 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T11:21:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-16T11:21:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-16T11:21:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000102514 s, 5.0 MB/s 2024-09-16T11:21:56.811 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T11:21:56.868 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-09-16T11:21:56.924 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T11:21:56.924 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:21:56.924 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T11:21:56.925 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:21:56.925 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:21:56.925 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-09-16 11:21:42.369621597 +0000 2024-09-16T11:21:56.925 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-09-16 11:19:11.524972524 +0000 2024-09-16T11:21:56.925 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-09-16 11:19:11.524972524 +0000 2024-09-16T11:21:56.925 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-09-16 11:19:11.524972524 +0000 2024-09-16T11:21:56.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T11:21:56.988 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-09-16T11:21:56.988 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-09-16T11:21:56.988 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.9556e-05 s, 5.1 MB/s 2024-09-16T11:21:56.990 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T11:21:57.046 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:21:57.046 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:21:57.046 INFO:tasks.ceph:Generating config... 2024-09-16T11:21:57.047 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.048 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.049 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.049 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.049 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.049 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.049 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.049 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T11:21:57.049 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T11:21:57.049 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T11:21:57.049 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T11:21:57.049 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T11:21:57.049 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T11:21:57.049 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T11:21:57.049 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T11:21:57.050 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T11:21:57.050 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T11:21:57.050 INFO:tasks.ceph:[osd] osd objectstore = filestore 2024-09-16T11:21:57.050 INFO:tasks.ceph:[osd] osd sloppy crc = True 2024-09-16T11:21:57.050 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-16T11:21:57.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.131 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.280 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.53')] 2024-09-16T11:21:57.281 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.53', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T11:21:57.281 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:57.281 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:21:57.336 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.53 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:21:57.413 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:21:57.413 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: generated fsid bfe848d9-9b22-4e38-923f-a2004deef198 2024-09-16T11:21:57.413 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = octopus 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 0 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:fsid bfe848d9-9b22-4e38-923f-a2004deef198 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:last_changed 2024-09-16T11:21:57.412194+0000 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:created 2024-09-16T11:21:57.412194+0000 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:min_mon_release 15 (octopus) 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:election_strategy: 1 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-09-16T11:21:57.414 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-16T11:21:57.415 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:21:57.471 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID bfe848d9-9b22-4e38-923f-a2004deef198... 2024-09-16T11:21:57.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T11:21:57.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-09-16T11:21:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = 172.21.15.53 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: mon client directed command retry = 5 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on pool no app = False 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: ms inject socket failures = 5000 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = bfe848d9-9b22-4e38-923f-a2004deef198 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.559 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-09-16T11:21:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = filestore 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: osd sloppy crc = True 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-09-16T11:21:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:21:57.564 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: debug rbd = 20 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: debug rbd_mirror = 15 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: log to stderr = False 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: rbd default features = 125 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.0] 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.1] 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.2] 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.3] 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.4] 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.566 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.5] 2024-09-16T11:21:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.567 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.6] 2024-09-16T11:21:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.567 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.568 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-09-16T11:21:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.53 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on pool no app = False 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = bfe848d9-9b22-4e38-923f-a2004deef198 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.571 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-09-16T11:21:57.572 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.573 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:21:57.573 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.573 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-09-16T11:21:57.573 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.573 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = filestore 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: osd sloppy crc = True 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-09-16T11:21:57.574 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-16T11:21:57.575 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.576 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-09-16T11:21:57.577 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd = 20 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd_mirror = 15 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: log to stderr = False 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: rbd default features = 125 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.578 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:21:57.579 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:21:57.580 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-16T11:21:57.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.672 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T11:21:57.672 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:57.672 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-16T11:21:57.688 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:57.688 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-16T11:21:57.743 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:21:57.743 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:57.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T11:21:57.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.818 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:21:57.818 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:21:57.874 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:21:57.874 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:21:57.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T11:21:57.874 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T11:21:57.915 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:21:57.915 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:21:57.973 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T11:21:57.974 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T11:21:57.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T11:21:58.023 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T11:21:58.025 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T11:21:58.025 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T11:21:58.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-16T11:21:58.065 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-16T11:21:58.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T11:21:58.133 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T11:21:58.147 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T11:21:58.186 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T11:21:58.199 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T11:21:58.278 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T11:21:58.290 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T11:21:58.368 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T11:21:58.381 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T11:21:58.459 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T11:21:58.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T11:21:58.549 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T11:21:58.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T11:21:58.640 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T11:21:58.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T11:21:58.731 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T11:21:58.745 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T11:21:58.745 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:21:58.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-16T11:21:58.774 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:21:58.774 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-16T11:21:58.774 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:21:58.774 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T11:21:58.842 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:21:58.843 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:21:58.875 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-16T11:21:58.877 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-16T11:21:58.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-16T11:21:58.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-16T11:21:59.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-16T11:21:59.091 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:21:59.091 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-16T11:21:59.091 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:21:59.091 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T11:21:59.156 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:21:59.157 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:21:59.188 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-16T11:21:59.190 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-16T11:21:59.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-16T11:21:59.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-16T11:21:59.333 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-16T11:21:59.399 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:21:59.399 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-16T11:21:59.399 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:21:59.399 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:21:59.465 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:21:59.466 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:21:59.498 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-09-16T11:21:59.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-09-16T11:21:59.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-16T11:21:59.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-16T11:21:59.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:21:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.745+0000 7f893af2f440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-16T11:21:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.745+0000 7f893af2f440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-16T11:21:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.749+0000 7f893af2f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:21:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.749+0000 7f893af2f440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-16T11:21:59.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.807+0000 7f893af2f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:21:59.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.807+0000 7f893af2f440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:21:59.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.807+0000 7f893af2f440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:21:59.816 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-16T11:21:59.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:21:59.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.921+0000 7f4eda436440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-16T11:21:59.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.922+0000 7f4eda436440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-16T11:21:59.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.926+0000 7f4eda436440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:21:59.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.926+0000 7f4eda436440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-16T11:21:59.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.982+0000 7f4eda436440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:21:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.982+0000 7f4eda436440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:21:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:21:59.982+0000 7f4eda436440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:21:59.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-16T11:22:00.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:22:00.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.136+0000 7efcb6986440 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-16T11:22:00.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.136+0000 7efcb6986440 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-16T11:22:00.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.140+0000 7efcb6986440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:00.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.140+0000 7efcb6986440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-16T11:22:00.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.197+0000 7efcb6986440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:00.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.197+0000 7efcb6986440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:00.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:00.197+0000 7efcb6986440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:00.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-16T11:22:00.272 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T11:22:00.273 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:00.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-16T11:22:00.338 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:00.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-16T11:22:00.404 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:00.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-16T11:22:00.469 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:00.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-16T11:22:00.535 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:00.535 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-16T11:22:00.591 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.591 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-16T11:22:00.611 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.611 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-16T11:22:00.668 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.668 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-16T11:22:00.726 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.726 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-16T11:22:00.784 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.785 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-16T11:22:00.842 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.842 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-16T11:22:00.900 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.900 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-16T11:22:00.956 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:00.957 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-16T11:22:01.014 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T11:22:01.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-16T11:22:01.038 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-09-16T11:22:01.038 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmgrVWARAA7tXZP0IqKYc54/oDdvJDwQ== 2024-09-16T11:22:01.038 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDXFOhmrJF6LBAALdWePN2r7wUjdl15RAhPIA== 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDXFOhmZdL+NhAAuTbRIlQEOX28NxXU5CmHoA== 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDYFOhmm20xCBAAfTmNvRfB6grFRAh6XwKXVQ== 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmH5TTAxAACbLEcQhFIHvUzjK0O7AUVA== 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmuwjiBxAAnOaAlZ/fpk1FhWJP1QpOMQ== 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.0] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmNlYNCxAAGFXXBC1jmxLj4PtF0gJzQQ== 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.1] 2024-09-16T11:22:01.039 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhm826HEBAA1GAcYi+bDOlzbgbaMBTUTA== 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.2] 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmnfTvFRAA9admVqKnXkHD/lQUAIcvPg== 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.3] 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmt+1VGxAA3QekYr5JwiHNEUQqXYM7JQ== 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.4] 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhm9VW2IBAAYSHyLyWI582uSxSv2jyuQw== 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.5] 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhm3dATJhAA6svE+KuF2qEQdh8DfG0brQ== 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.6] 2024-09-16T11:22:01.040 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDWFOhmiLaRKxAA9dN3lPFdg5Mfrxy1H22avA== 2024-09-16T11:22:01.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T11:22:01.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:22:01.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:22:01.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:22:01.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.373 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:01.724 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T11:22:01.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-16T11:22:01.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-16T11:22:01.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-16T11:22:01.867 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T11:22:01.923 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-16T11:22:01.924 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-16T11:22:01.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-16T11:22:01.966 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-16T11:22:01.966 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-16T11:22:01.966 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-16T11:22:01.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-16T11:22:01.969 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-16T11:22:01.969 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T11:22:01.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-16T11:22:02.140 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-09-16T11:22:02.150 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-16T11:22:02.151 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:02.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-16T11:22:02.176 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:02.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-16T11:22:02.242 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:02.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-16T11:22:02.307 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster cluster1 osd new 9d8548df-278f-4b8b-a7c0-4faddbf572f1 0 2024-09-16T11:22:02.340 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:02.338+0000 7fb940d67ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T11:22:02.381 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:02.379+0000 7fb940d67ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T11:22:02.434 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:02.432+0000 7fb940d67ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T11:22:02.513 INFO:teuthology.orchestra.run.smithi053.stdout:0 2024-09-16T11:22:02.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster cluster1 osd new f9d9193d-5e46-49cc-ab26-34618db4a07d 1 2024-09-16T11:22:02.580 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:02.578+0000 7fb940d67ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:22:02.695 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:02.693+0000 7fb940d67ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T11:22:02.739 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-09-16T11:22:02.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster cluster1 osd new 635ae354-dc24-4826-9d97-3c5c80914da2 2 2024-09-16T11:22:02.963 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-09-16T11:22:02.974 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-16T11:22:02.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-16T11:22:02.976 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-16T11:22:02.976 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-16T11:22:02.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-16T11:22:02.978 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-16T11:22:02.978 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-16T11:22:02.979 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-16T11:22:02.981 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-16T11:22:02.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:22:02.995 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:02.993+0000 7fb940d67ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T11:22:03.056 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:03.054+0000 7fdd76323440 -1 Falling back to public interface 2024-09-16T11:22:03.067 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:03.065+0000 7fdd76323440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:03.067 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:03.065+0000 7fdd76323440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:03.067 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:03.065+0000 7fdd76323440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:03.069 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:03.067+0000 7fb940d67ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T11:22:03.073 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:03.071+0000 7fdd76323440 -1 osd.1 0 log_to_monitors true 2024-09-16T11:22:03.075 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:03.074+0000 7fd885816440 -1 Falling back to public interface 2024-09-16T11:22:03.080 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:03.078+0000 7f4430f2b440 -1 Falling back to public interface 2024-09-16T11:22:03.090 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:03.088+0000 7fd885816440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:03.090 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:03.088+0000 7fd885816440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:03.090 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:03.088+0000 7fd885816440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:03.096 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:03.095+0000 7fd885816440 -1 osd.0 0 log_to_monitors true 2024-09-16T11:22:03.101 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:03.099+0000 7f4430f2b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:03.101 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:03.099+0000 7f4430f2b440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:03.101 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:03.099+0000 7f4430f2b440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:03.114 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:03.112+0000 7f4430f2b440 -1 osd.2 0 log_to_monitors true 2024-09-16T11:22:03.225 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:03.224+0000 7fb940d67ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T11:22:03.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-09-16T11:22:03.294+0000 7f77d4835640 0 --2- 172.21.15.53:0/2306267889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77d00a5360 0x7f77d00a6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T11:22:03.357 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:03.358 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:22:02.960807+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:03.368 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-16T11:22:03.368 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T11:22:03.671 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:22:03.829 INFO:teuthology.misc.health.smithi053.stdout: 2024-09-16T11:22:03.829 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":5,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:22:02.960807+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:03.839 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T11:22:04.225 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.223+0000 7fb940d67ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T11:22:04.277 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.275+0000 7fb940d67ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T11:22:04.659 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.657+0000 7fb940d67ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T11:22:04.744 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.742+0000 7fb940d67ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T11:22:04.836 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.834+0000 7fb940d67ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:22:04.882 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.881+0000 7fb940d67ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T11:22:04.980 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:04.979+0000 7fb940d67ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T11:22:05.035 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.033+0000 7fb940d67ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T11:22:05.058 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:05.057+0000 7fd8807f7640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T11:22:05.059 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:05.057+0000 7f442c1f7640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T11:22:05.059 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:05.057+0000 7fdd715f7640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T11:22:05.060 INFO:tasks.ceph.cluster1.osd.2.smithi053.stderr:2024-09-16T11:22:05.058+0000 7f44289f0640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:22:05.060 INFO:tasks.ceph.cluster1.osd.1.smithi053.stderr:2024-09-16T11:22:05.058+0000 7fdd6ddf0640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:22:05.060 INFO:tasks.ceph.cluster1.osd.0.smithi053.stderr:2024-09-16T11:22:05.058+0000 7fd87cff0640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:22:05.082 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.080+0000 7fb940d67ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T11:22:05.179 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.177+0000 7fb940d67ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T11:22:05.230 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.228+0000 7fb940d67ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T11:22:05.282 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.281+0000 7fb940d67ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T11:22:05.352 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.350+0000 7fb940d67ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T11:22:05.396 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.394+0000 7fb940d67ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T11:22:05.441 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:05.439+0000 7fb940d67ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T11:22:06.085 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:06.083+0000 7fb940d67ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T11:22:07.097 INFO:tasks.ceph.cluster1.mgr.x.smithi053.stderr:2024-09-16T11:22:07.095+0000 7fb925c78640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T11:22:10.142 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:22:10.420 INFO:teuthology.misc.health.smithi053.stdout: 2024-09-16T11:22:10.420 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":11,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:22:10.100279+0000","last_up_change":"2024-09-16T11:22:06.056486+0000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:08.131565+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":28635},{"type":"v1","addr":"172.21.15.53:6809","nonce":28635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":28635},{"type":"v1","addr":"172.21.15.53:6811","nonce":28635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":28635},{"type":"v1","addr":"172.21.15.53:6815","nonce":28635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":28635},{"type":"v1","addr":"172.21.15.53:6813","nonce":28635}]},"public_addr":"172.21.15.53:6809/28635","cluster_addr":"172.21.15.53:6811/28635","heartbeat_back_addr":"172.21.15.53:6815/28635","heartbeat_front_addr":"172.21.15.53:6813/28635","state":["exists","up"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":28626},{"type":"v1","addr":"172.21.15.53:6801","nonce":28626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":28626},{"type":"v1","addr":"172.21.15.53:6803","nonce":28626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":28626},{"type":"v1","addr":"172.21.15.53:6807","nonce":28626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":28626},{"type":"v1","addr":"172.21.15.53:6805","nonce":28626}]},"public_addr":"172.21.15.53:6801/28626","cluster_addr":"172.21.15.53:6803/28626","heartbeat_back_addr":"172.21.15.53:6807/28626","heartbeat_front_addr":"172.21.15.53:6805/28626","state":["exists","up"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":28638},{"type":"v1","addr":"172.21.15.53:6817","nonce":28638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":28638},{"type":"v1","addr":"172.21.15.53:6819","nonce":28638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":28638},{"type":"v1","addr":"172.21.15.53:6823","nonce":28638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":28638},{"type":"v1","addr":"172.21.15.53:6821","nonce":28638}]},"public_addr":"172.21.15.53:6817/28638","cluster_addr":"172.21.15.53:6819/28638","heartbeat_back_addr":"172.21.15.53:6823/28638","heartbeat_front_addr":"172.21.15.53:6821/28638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107422+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.097413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:10.432 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T11:22:10.432 INFO:tasks.ceph:Creating RBD pool 2024-09-16T11:22:10.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-16T11:22:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'rbd' created 2024-09-16T11:22:11.125 DEBUG:teuthology.orchestra.run.smithi053:> rbd --cluster cluster1 pool init rbd 2024-09-16T11:22:14.125 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-16T11:22:14.126 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-16T11:22:14.126 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-16T11:22:14.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-16T11:22:14.126 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T11:22:14.458 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:14.469 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":28290},{"type":"v1","addr":"172.21.15.53:6825","nonce":28290}]},"active_addr":"172.21.15.53:6825/28290","active_change":"2024-09-16T11:22:06.085391+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1382950285}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2872763188}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":4157027657}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1957510389}]}]} 2024-09-16T11:22:14.474 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-16T11:22:14.474 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-16T11:22:14.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:22:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:14.747 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":15,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:22:14.112544+0000","last_up_change":"2024-09-16T11:22:06.056486+0000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:08.131565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:10.731960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":28635},{"type":"v1","addr":"172.21.15.53:6809","nonce":28635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":28635},{"type":"v1","addr":"172.21.15.53:6811","nonce":28635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":28635},{"type":"v1","addr":"172.21.15.53:6815","nonce":28635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":28635},{"type":"v1","addr":"172.21.15.53:6813","nonce":28635}]},"public_addr":"172.21.15.53:6809/28635","cluster_addr":"172.21.15.53:6811/28635","heartbeat_back_addr":"172.21.15.53:6815/28635","heartbeat_front_addr":"172.21.15.53:6813/28635","state":["exists","up"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":28626},{"type":"v1","addr":"172.21.15.53:6801","nonce":28626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":28626},{"type":"v1","addr":"172.21.15.53:6803","nonce":28626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":28626},{"type":"v1","addr":"172.21.15.53:6807","nonce":28626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":28626},{"type":"v1","addr":"172.21.15.53:6805","nonce":28626}]},"public_addr":"172.21.15.53:6801/28626","cluster_addr":"172.21.15.53:6803/28626","heartbeat_back_addr":"172.21.15.53:6807/28626","heartbeat_front_addr":"172.21.15.53:6805/28626","state":["exists","up"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":28638},{"type":"v1","addr":"172.21.15.53:6817","nonce":28638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":28638},{"type":"v1","addr":"172.21.15.53:6819","nonce":28638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":28638},{"type":"v1","addr":"172.21.15.53:6823","nonce":28638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":28638},{"type":"v1","addr":"172.21.15.53:6821","nonce":28638}]},"public_addr":"172.21.15.53:6817/28638","cluster_addr":"172.21.15.53:6819/28638","heartbeat_back_addr":"172.21.15.53:6823/28638","heartbeat_front_addr":"172.21.15.53:6821/28638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107422+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.097413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:14.759 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-16T11:22:14.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:22:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:15.030 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":15,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:22:14.112544+0000","last_up_change":"2024-09-16T11:22:06.056486+0000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:08.131565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:10.731960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":28635},{"type":"v1","addr":"172.21.15.53:6809","nonce":28635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":28635},{"type":"v1","addr":"172.21.15.53:6811","nonce":28635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":28635},{"type":"v1","addr":"172.21.15.53:6815","nonce":28635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":28635},{"type":"v1","addr":"172.21.15.53:6813","nonce":28635}]},"public_addr":"172.21.15.53:6809/28635","cluster_addr":"172.21.15.53:6811/28635","heartbeat_back_addr":"172.21.15.53:6815/28635","heartbeat_front_addr":"172.21.15.53:6813/28635","state":["exists","up"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":28626},{"type":"v1","addr":"172.21.15.53:6801","nonce":28626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":28626},{"type":"v1","addr":"172.21.15.53:6803","nonce":28626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":28626},{"type":"v1","addr":"172.21.15.53:6807","nonce":28626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":28626},{"type":"v1","addr":"172.21.15.53:6805","nonce":28626}]},"public_addr":"172.21.15.53:6801/28626","cluster_addr":"172.21.15.53:6803/28626","heartbeat_back_addr":"172.21.15.53:6807/28626","heartbeat_front_addr":"172.21.15.53:6805/28626","state":["exists","up"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":28638},{"type":"v1","addr":"172.21.15.53:6817","nonce":28638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":28638},{"type":"v1","addr":"172.21.15.53:6819","nonce":28638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":28638},{"type":"v1","addr":"172.21.15.53:6823","nonce":28638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":28638},{"type":"v1","addr":"172.21.15.53:6821","nonce":28638}]},"public_addr":"172.21.15.53:6817/28638","cluster_addr":"172.21.15.53:6819/28638","heartbeat_back_addr":"172.21.15.53:6823/28638","heartbeat_front_addr":"172.21.15.53:6821/28638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107422+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.097413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:15.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-16T11:22:15.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-16T11:22:15.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-16T11:22:15.176 INFO:teuthology.orchestra.run.smithi053.stdout:34359738371 2024-09-16T11:22:15.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T11:22:15.195 INFO:teuthology.orchestra.run.smithi053.stdout:34359738371 2024-09-16T11:22:15.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T11:22:15.196 INFO:teuthology.orchestra.run.smithi053.stdout:34359738371 2024-09-16T11:22:15.196 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T11:22:15.460 INFO:teuthology.orchestra.run.smithi053.stdout:34359738370 2024-09-16T11:22:15.472 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.1 2024-09-16T11:22:15.535 INFO:teuthology.orchestra.run.smithi053.stdout:34359738370 2024-09-16T11:22:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:34359738370 2024-09-16T11:22:15.547 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2024-09-16T11:22:15.554 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.2 2024-09-16T11:22:16.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T11:22:16.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T11:22:16.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T11:22:16.766 INFO:teuthology.orchestra.run.smithi053.stdout:34359738371 2024-09-16T11:22:16.780 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2024-09-16T11:22:16.780 DEBUG:teuthology.parallel:result is None 2024-09-16T11:22:16.840 INFO:teuthology.orchestra.run.smithi053.stdout:34359738371 2024-09-16T11:22:16.853 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2024-09-16T11:22:16.853 DEBUG:teuthology.parallel:result is None 2024-09-16T11:22:16.877 INFO:teuthology.orchestra.run.smithi053.stdout:34359738371 2024-09-16T11:22:16.889 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2024-09-16T11:22:16.889 DEBUG:teuthology.parallel:result is None 2024-09-16T11:22:16.890 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T11:22:16.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:22:17.205 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:17.206 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:22:17.217 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T11:22:17.044779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2389796,"kb_used_data":278649176,"kb_used_omap":298,"kb_used_meta":0,"kb_avail":278649052,"statfs":{"total":287783780352,"available":285336629248,"internally_reserved":126976,"allocated":285336756224,"data_stored":285336756224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":305734,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.935204"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119980+0000","last_change":"2024-09-16T11:22:14.120245+0000","last_active":"2024-09-16T11:22:14.119980+0000","last_peered":"2024-09-16T11:22:14.119980+0000","last_clean":"2024-09-16T11:22:14.119980+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:22:14.119980+0000","last_undegraded":"2024-09-16T11:22:14.119980+0000","last_fullsized":"2024-09-16T11:22:14.119980+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:50.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119638+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:22:14.119638+0000","last_peered":"2024-09-16T11:22:14.119638+0000","last_clean":"2024-09-16T11:22:14.119638+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:22:14.119638+0000","last_undegraded":"2024-09-16T11:22:14.119638+0000","last_fullsized":"2024-09-16T11:22:14.119638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:07.810548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119683+0000","last_change":"2024-09-16T11:22:14.120010+0000","last_active":"2024-09-16T11:22:14.119683+0000","last_peered":"2024-09-16T11:22:14.119683+0000","last_clean":"2024-09-16T11:22:14.119683+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:22:14.119683+0000","last_undegraded":"2024-09-16T11:22:14.119683+0000","last_fullsized":"2024-09-16T11:22:14.119683+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:43.889549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119157+0000","last_change":"2024-09-16T11:22:14.119525+0000","last_active":"2024-09-16T11:22:14.119157+0000","last_peered":"2024-09-16T11:22:14.119157+0000","last_clean":"2024-09-16T11:22:14.119157+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:22:14.119157+0000","last_undegraded":"2024-09-16T11:22:14.119157+0000","last_fullsized":"2024-09-16T11:22:14.119157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:21.893309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.118460+0000","last_change":"2024-09-16T11:22:14.118460+0000","last_active":"2024-09-16T11:22:14.118460+0000","last_peered":"2024-09-16T11:22:14.118460+0000","last_clean":"2024-09-16T11:22:14.118460+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:22:14.118460+0000","last_undegraded":"2024-09-16T11:22:14.118460+0000","last_fullsized":"2024-09-16T11:22:14.118460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:29:26.098123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.115872+0000","last_change":"2024-09-16T11:22:14.116075+0000","last_active":"2024-09-16T11:22:14.115872+0000","last_peered":"2024-09-16T11:22:14.115872+0000","last_clean":"2024-09-16T11:22:14.115872+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:22:14.115872+0000","last_undegraded":"2024-09-16T11:22:14.115872+0000","last_fullsized":"2024-09-16T11:22:14.115872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:59.034856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.115926+0000","last_change":"2024-09-16T11:22:14.116152+0000","last_active":"2024-09-16T11:22:14.115926+0000","last_peered":"2024-09-16T11:22:14.115926+0000","last_clean":"2024-09-16T11:22:14.115926+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:22:14.115926+0000","last_undegraded":"2024-09-16T11:22:14.115926+0000","last_fullsized":"2024-09-16T11:22:14.115926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:26.664156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119370+0000","last_change":"2024-09-16T11:22:14.119568+0000","last_active":"2024-09-16T11:22:14.119370+0000","last_peered":"2024-09-16T11:22:14.119370+0000","last_clean":"2024-09-16T11:22:14.119370+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:22:14.119370+0000","last_undegraded":"2024-09-16T11:22:14.119370+0000","last_fullsized":"2024-09-16T11:22:14.119370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:46.653548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":79,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T11:22:13.113224+0000","last_change":"2024-09-16T11:22:10.960454+0000","last_active":"2024-09-16T11:22:13.113224+0000","last_peered":"2024-09-16T11:22:13.113224+0000","last_clean":"2024-09-16T11:22:13.113224+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:22:13.113224+0000","last_undegraded":"2024-09-16T11:22:13.113224+0000","last_fullsized":"2024-09-16T11:22:13.113224+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_clean_scrub_stamp":"2024-09-16T11:22:09.097683+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:09.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795752,"kb_used_data":92883888,"kb_used_omap":60,"kb_used_meta":0,"kb_avail":92883864,"statfs":{"total":95927926784,"available":95113076736,"internally_reserved":24576,"allocated":95113101312,"data_stored":95113101312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61683,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797028,"kb_used_data":92882644,"kb_used_omap":128,"kb_used_meta":0,"kb_avail":92882588,"statfs":{"total":95927926784,"available":95111770112,"internally_reserved":57344,"allocated":95111827456,"data_stored":95111827456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131379,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797016,"kb_used_data":92882644,"kb_used_omap":110,"kb_used_meta":0,"kb_avail":92882600,"statfs":{"total":95927926784,"available":95111782400,"internally_reserved":45056,"allocated":95111827456,"data_stored":95111827456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112672,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:22:17.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:22:17.484 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:17.485 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:22:17.496 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-16T11:22:17.044779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2389796,"kb_used_data":278649176,"kb_used_omap":298,"kb_used_meta":0,"kb_avail":278649052,"statfs":{"total":287783780352,"available":285336629248,"internally_reserved":126976,"allocated":285336756224,"data_stored":285336756224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":305734,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.935204"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119980+0000","last_change":"2024-09-16T11:22:14.120245+0000","last_active":"2024-09-16T11:22:14.119980+0000","last_peered":"2024-09-16T11:22:14.119980+0000","last_clean":"2024-09-16T11:22:14.119980+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:22:14.119980+0000","last_undegraded":"2024-09-16T11:22:14.119980+0000","last_fullsized":"2024-09-16T11:22:14.119980+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:50.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119638+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:22:14.119638+0000","last_peered":"2024-09-16T11:22:14.119638+0000","last_clean":"2024-09-16T11:22:14.119638+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:22:14.119638+0000","last_undegraded":"2024-09-16T11:22:14.119638+0000","last_fullsized":"2024-09-16T11:22:14.119638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:07.810548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119683+0000","last_change":"2024-09-16T11:22:14.120010+0000","last_active":"2024-09-16T11:22:14.119683+0000","last_peered":"2024-09-16T11:22:14.119683+0000","last_clean":"2024-09-16T11:22:14.119683+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:22:14.119683+0000","last_undegraded":"2024-09-16T11:22:14.119683+0000","last_fullsized":"2024-09-16T11:22:14.119683+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:43.889549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119157+0000","last_change":"2024-09-16T11:22:14.119525+0000","last_active":"2024-09-16T11:22:14.119157+0000","last_peered":"2024-09-16T11:22:14.119157+0000","last_clean":"2024-09-16T11:22:14.119157+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:22:14.119157+0000","last_undegraded":"2024-09-16T11:22:14.119157+0000","last_fullsized":"2024-09-16T11:22:14.119157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:21.893309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.118460+0000","last_change":"2024-09-16T11:22:14.118460+0000","last_active":"2024-09-16T11:22:14.118460+0000","last_peered":"2024-09-16T11:22:14.118460+0000","last_clean":"2024-09-16T11:22:14.118460+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:22:14.118460+0000","last_undegraded":"2024-09-16T11:22:14.118460+0000","last_fullsized":"2024-09-16T11:22:14.118460+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:29:26.098123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.115872+0000","last_change":"2024-09-16T11:22:14.116075+0000","last_active":"2024-09-16T11:22:14.115872+0000","last_peered":"2024-09-16T11:22:14.115872+0000","last_clean":"2024-09-16T11:22:14.115872+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:22:14.115872+0000","last_undegraded":"2024-09-16T11:22:14.115872+0000","last_fullsized":"2024-09-16T11:22:14.115872+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:59.034856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.115926+0000","last_change":"2024-09-16T11:22:14.116152+0000","last_active":"2024-09-16T11:22:14.115926+0000","last_peered":"2024-09-16T11:22:14.115926+0000","last_clean":"2024-09-16T11:22:14.115926+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:22:14.115926+0000","last_undegraded":"2024-09-16T11:22:14.115926+0000","last_fullsized":"2024-09-16T11:22:14.115926+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:26.664156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:14.119370+0000","last_change":"2024-09-16T11:22:14.119568+0000","last_active":"2024-09-16T11:22:14.119370+0000","last_peered":"2024-09-16T11:22:14.119370+0000","last_clean":"2024-09-16T11:22:14.119370+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:22:14.119370+0000","last_undegraded":"2024-09-16T11:22:14.119370+0000","last_fullsized":"2024-09-16T11:22:14.119370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:46.653548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":79,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-16T11:22:13.113224+0000","last_change":"2024-09-16T11:22:10.960454+0000","last_active":"2024-09-16T11:22:13.113224+0000","last_peered":"2024-09-16T11:22:13.113224+0000","last_clean":"2024-09-16T11:22:13.113224+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:22:13.113224+0000","last_undegraded":"2024-09-16T11:22:13.113224+0000","last_fullsized":"2024-09-16T11:22:13.113224+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_clean_scrub_stamp":"2024-09-16T11:22:09.097683+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:09.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795752,"kb_used_data":92883888,"kb_used_omap":60,"kb_used_meta":0,"kb_avail":92883864,"statfs":{"total":95927926784,"available":95113076736,"internally_reserved":24576,"allocated":95113101312,"data_stored":95113101312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61683,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797028,"kb_used_data":92882644,"kb_used_omap":128,"kb_used_meta":0,"kb_avail":92882588,"statfs":{"total":95927926784,"available":95111770112,"internally_reserved":57344,"allocated":95111827456,"data_stored":95111827456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131379,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797016,"kb_used_data":92882644,"kb_used_omap":110,"kb_used_meta":0,"kb_avail":92882600,"statfs":{"total":95927926784,"available":95111782400,"internally_reserved":45056,"allocated":95111827456,"data_stored":95111827456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112672,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:22:17.496 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T11:22:17.497 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-16T11:22:17.497 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-16T11:22:17.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-16T11:22:17.798 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:17.799 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T11:22:17.812 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-16T11:22:17.812 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T11:22:17.824 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-16T11:22:17.824 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T11:22:17.825 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907185', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOe7lMxi92VYCpCtte6dzjkYKO5OGVDJ7IZhNUiqKzRWoeCIfgcJG4poQUlmzBExppekzihp9JzY978AaCEVF+Q=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDMCaz0LRwVAGDTbGa15b9GXxNEYnUK+l3jpCRHQyjqf7CNnXJKrdfB7vxJ/LVkYxdAuKcPrxoCK843457LUB6I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_journal.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T11:22:17.825 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T11:22:17.828 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T11:22:17.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:22:17.884 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T11:22:17.916 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:17.916 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T11:22:17.975 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T11:22:17.976 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-09-16T11:22:18.034 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T11:22:18.034 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:22:18.034 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T11:22:18.034 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:22:18.035 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:22:18.035 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 11:21:50.958841653 +0000 2024-09-16T11:22:18.035 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 11:19:07.098376097 +0000 2024-09-16T11:22:18.035 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 11:19:07.098376097 +0000 2024-09-16T11:22:18.035 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 11:19:07.098376097 +0000 2024-09-16T11:22:18.035 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T11:22:18.102 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T11:22:18.103 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T11:22:18.103 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.00022769 s, 2.2 MB/s 2024-09-16T11:22:18.104 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T11:22:18.163 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 11:21:50.959841638 +0000 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 11:19:07.440371243 +0000 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 11:19:07.440371243 +0000 2024-09-16T11:22:18.221 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 11:19:07.440371243 +0000 2024-09-16T11:22:18.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T11:22:18.287 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T11:22:18.287 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T11:22:18.287 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000229384 s, 2.2 MB/s 2024-09-16T11:22:18.289 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T11:22:18.346 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-09-16T11:22:18.403 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T11:22:18.403 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:22:18.403 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T11:22:18.403 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:22:18.404 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:22:18.404 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 11:21:50.959841638 +0000 2024-09-16T11:22:18.404 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 11:19:07.765366630 +0000 2024-09-16T11:22:18.404 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 11:19:07.765366630 +0000 2024-09-16T11:22:18.404 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 11:19:07.765366630 +0000 2024-09-16T11:22:18.404 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T11:22:18.468 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T11:22:18.468 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T11:22:18.468 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000190872 s, 2.7 MB/s 2024-09-16T11:22:18.469 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T11:22:18.527 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-09-16T11:22:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T11:22:18.582 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T11:22:18.582 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T11:22:18.582 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T11:22:18.582 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T11:22:18.582 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-16 11:21:50.959841638 +0000 2024-09-16T11:22:18.583 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-16 11:19:08.094361961 +0000 2024-09-16T11:22:18.583 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-16 11:19:08.094361961 +0000 2024-09-16T11:22:18.583 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-16 11:19:08.094361961 +0000 2024-09-16T11:22:18.583 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T11:22:18.647 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-16T11:22:18.647 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-16T11:22:18.647 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000197605 s, 2.6 MB/s 2024-09-16T11:22:18.649 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T11:22:18.706 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:22:18.706 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:22:18.706 INFO:tasks.ceph:Generating config... 2024-09-16T11:22:18.707 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.708 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:18.709 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:18.709 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T11:22:18.709 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T11:22:18.709 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T11:22:18.709 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T11:22:18.710 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T11:22:18.710 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T11:22:18.710 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T11:22:18.710 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T11:22:18.710 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T11:22:18.710 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T11:22:18.710 INFO:tasks.ceph:[osd] osd objectstore = filestore 2024-09-16T11:22:18.710 INFO:tasks.ceph:[osd] osd sloppy crc = True 2024-09-16T11:22:18.710 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-16T11:22:18.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-16T11:22:18.783 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.keyring 2024-09-16T11:22:18.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-16T11:22:18.865 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T11:22:18.930 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.204')] 2024-09-16T11:22:18.930 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.204', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T11:22:18.931 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:18.931 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:22:18.985 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.204 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: generated fsid 8fbb69d7-ce8e-4174-822f-8bd0c8eba787 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:setting min_mon_release = octopus 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:epoch 0 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:fsid 8fbb69d7-ce8e-4174-822f-8bd0c8eba787 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:last_changed 2024-09-16T11:22:19.060179+0000 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:created 2024-09-16T11:22:19.060179+0000 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:min_mon_release 15 (octopus) 2024-09-16T11:22:19.061 INFO:teuthology.orchestra.run.smithi204.stdout:election_strategy: 1 2024-09-16T11:22:19.062 INFO:teuthology.orchestra.run.smithi204.stdout:0: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.a 2024-09-16T11:22:19.062 INFO:teuthology.orchestra.run.smithi204.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-16T11:22:19.063 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T11:22:19.117 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 8fbb69d7-ce8e-4174-822f-8bd0c8eba787... 2024-09-16T11:22:19.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T11:22:19.122 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T11:22:19.173 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-09-16T11:22:19.173 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-09-16T11:22:19.173 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:22:19.173 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-09-16T11:22:19.173 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.173 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.174 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:22:19.175 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.176 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-09-16T11:22:19.177 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = 172.21.15.204 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: mon client directed command retry = 5 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on pool no app = False 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: ms inject socket failures = 5000 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 8fbb69d7-ce8e-4174-822f-8bd0c8eba787 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.178 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.179 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = filestore 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: osd sloppy crc = True 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-09-16T11:22:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-09-16T11:22:19.181 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:22:19.182 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:22:19.183 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: debug rbd = 20 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: debug rbd_mirror = 15 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: log to stderr = False 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: rbd default features = 125 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.0] 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.1] 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.184 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.2] 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.3] 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.4] 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.185 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.5] 2024-09-16T11:22:19.186 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.186 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.186 INFO:teuthology.orchestra.run.smithi053.stdout:[client.mirror.6] 2024-09-16T11:22:19.186 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.186 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.201 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-09-16T11:22:19.201 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-09-16T11:22:19.201 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-09-16T11:22:19.202 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.203 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.204 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = 172.21.15.204 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-09-16T11:22:19.205 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on pool no app = False 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = 8fbb69d7-ce8e-4174-822f-8bd0c8eba787 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.206 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-09-16T11:22:19.207 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = filestore 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: osd sloppy crc = True 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-16T11:22:19.208 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-09-16T11:22:19.209 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-09-16T11:22:19.210 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd = 20 2024-09-16T11:22:19.211 INFO:teuthology.orchestra.run.smithi204.stdout: debug rbd_mirror = 15 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout: log to stderr = False 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout: rbd default features = 125 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.212 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2024-09-16T11:22:19.213 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.214 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.214 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2024-09-16T11:22:19.214 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T11:22:19.214 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T11:22:19.215 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-16T11:22:19.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-16T11:22:19.304 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T11:22:19.304 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:19.304 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-16T11:22:19.319 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:19.319 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-16T11:22:19.374 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:22:19.375 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:19.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T11:22:19.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T11:22:19.418 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:19.418 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:19.477 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:22:19.477 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:19.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T11:22:19.478 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T11:22:19.513 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:19.513 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:19.568 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T11:22:19.568 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T11:22:19.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T11:22:19.625 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T11:22:19.627 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T11:22:19.628 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T11:22:19.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-16T11:22:19.669 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-16T11:22:19.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T11:22:19.721 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T11:22:19.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T11:22:19.811 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T11:22:19.823 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T11:22:19.901 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T11:22:19.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T11:22:19.990 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T11:22:20.003 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T11:22:20.080 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T11:22:20.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T11:22:20.170 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T11:22:20.183 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T11:22:20.261 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T11:22:20.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T11:22:20.351 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T11:22:20.363 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T11:22:20.363 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T11:22:20.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-16T11:22:20.428 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:22:20.428 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-16T11:22:20.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:22:20.428 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:22:20.501 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:22:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:22:20.502 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:22:20.502 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:22:21.004 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-16T11:22:21.006 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-16T11:22:21.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-16T11:22:21.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-16T11:22:21.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-16T11:22:21.254 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:22:21.254 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-16T11:22:21.254 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:22:21.254 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T11:22:21.328 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:22:21.329 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:22:21.771 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-16T11:22:21.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-16T11:22:21.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-16T11:22:21.864 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-16T11:22:21.896 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-16T11:22:21.964 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T11:22:21.964 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-16T11:22:21.964 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:22:21.964 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T11:22:22.037 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T11:22:22.038 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T11:22:22.524 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-16T11:22:22.526 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-16T11:22:22.526 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-16T11:22:22.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-16T11:22:22.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:22.810 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:22.808+0000 7fc73e32b440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-16T11:22:22.814 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:22.813+0000 7fc73e32b440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-16T11:22:22.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:22.839+0000 7fc73e32b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:22.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:22.839+0000 7fc73e32b440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-16T11:22:23.089 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.087+0000 7fc73e32b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:23.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.090+0000 7fc73e32b440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:23.092 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.090+0000 7fc73e32b440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:23.121 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-16T11:22:23.150 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:23.235 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.234+0000 7faa43209440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-16T11:22:23.239 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.238+0000 7faa43209440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-16T11:22:23.267 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.265+0000 7faa43209440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:23.267 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.265+0000 7faa43209440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-16T11:22:23.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.484+0000 7faa43209440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:23.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.484+0000 7faa43209440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:23.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.484+0000 7faa43209440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:23.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-16T11:22:23.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:23.622 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.620+0000 7f384e11d440 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-16T11:22:23.622 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.620+0000 7f384e11d440 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-16T11:22:23.626 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.625+0000 7f384e11d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:23.626 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.625+0000 7f384e11d440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-16T11:22:23.812 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.810+0000 7f384e11d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:23.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.811+0000 7f384e11d440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:23.813 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-16T11:22:23.811+0000 7f384e11d440 -1 journal do_read_entry(4096): bad header magic 2024-09-16T11:22:23.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-16T11:22:23.885 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T11:22:23.885 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:23.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-16T11:22:23.917 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:23.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-16T11:22:23.950 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:23.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-16T11:22:24.016 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-16T11:22:24.080 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-09-16T11:22:24.080 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-16T11:22:24.099 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.099 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-16T11:22:24.135 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.136 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-16T11:22:24.191 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.191 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-16T11:22:24.246 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.246 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-16T11:22:24.302 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.302 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-16T11:22:24.358 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.358 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-16T11:22:24.414 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.414 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-16T11:22:24.469 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:24.470 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-16T11:22:24.524 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T11:22:24.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-16T11:22:24.590 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-09-16T11:22:24.590 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDrFOhmhyk8JRAAeEL63u8imIVSZbQVdnKm7g== 2024-09-16T11:22:24.590 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDuFOhmzE9DMBAA5mkZnxd2Gjo8bPvk9ZfcUQ== 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDvFOhmu8H/DRAAZmSu3L34EzQgDcHCltAPpw== 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDvFOhmgj0GJRAAtUtiRwQyDfy5KsSGcg+dwQ== 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDrFOhmv6ziJxAARybQpC4cmghMALf03zOBBg== 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror] 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDrFOhm46XtKhAA3aeXwwtQ0bir9ielpEghWA== 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.0] 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDrFOhmfftMMBAAfuzN79bhlil9+TemU2KBNA== 2024-09-16T11:22:24.591 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.1] 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDrFOhmipeqNRAAStLmyloziJI0QEvJvK3j1Q== 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.2] 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDrFOhmgv/8OhAAc7RTnbILybb5mofZixj4FA== 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.3] 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDsFOhmDq7DBBAAHYny61ittqoQsQB4wicfog== 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.4] 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDsFOhmArsgChAA1PbmH/tB2a5j63AZn4ofzg== 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.5] 2024-09-16T11:22:24.592 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDsFOhmdqiCDxAAkdmv/EhtcFtHAMe4dj/SWQ== 2024-09-16T11:22:24.593 INFO:teuthology.orchestra.run.smithi204.stdout:[client.mirror.6] 2024-09-16T11:22:24.593 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQDsFOhmxHndFBAAotUnbOOum/b95plMcUW62Q== 2024-09-16T11:22:24.594 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T11:22:24.680 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:22:24.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:22:24.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T11:22:24.819 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:24.868 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:24.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:24.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:25.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:25.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:25.114 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:25.164 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:25.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T11:22:25.262 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T11:22:25.262 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-16T11:22:25.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-16T11:22:25.379 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-16T11:22:25.403 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T11:22:25.458 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-16T11:22:25.459 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-16T11:22:25.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-16T11:22:25.501 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-16T11:22:25.501 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-16T11:22:25.501 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-16T11:22:25.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-16T11:22:25.504 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-16T11:22:25.504 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T11:22:25.504 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-16T11:22:25.665 INFO:teuthology.orchestra.run.smithi204.stderr:adjusted tunables profile to default 2024-09-16T11:22:25.667 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:25.666+0000 7f9203dc4ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T11:22:25.676 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-16T11:22:25.676 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:25.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-16T11:22:25.701 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:25.701 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-16T11:22:25.708 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:25.707+0000 7f9203dc4ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T11:22:25.757 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:25.755+0000 7f9203dc4ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T11:22:25.765 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:22:25.765 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-16T11:22:25.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new f01f38ee-93c2-4c2c-a776-70b9fb002a04 0 2024-09-16T11:22:25.850 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:25.848+0000 7f9203dc4ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:22:25.942 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:25.941+0000 7f9203dc4ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T11:22:26.037 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-09-16T11:22:26.048 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new 2b6c4ac5-2edf-4df3-974a-7f37ce290945 1 2024-09-16T11:22:26.179 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:26.177+0000 7f9203dc4ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T11:22:26.213 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-09-16T11:22:26.223 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:26.221+0000 7f9203dc4ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T11:22:26.224 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd new c21f06cf-02c3-4106-82d1-d9f01e320a2d 2 2024-09-16T11:22:26.324 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:26.322+0000 7f9203dc4ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T11:22:26.390 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-09-16T11:22:26.401 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-16T11:22:26.401 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-16T11:22:26.403 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-16T11:22:26.403 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-16T11:22:26.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-16T11:22:26.405 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-16T11:22:26.405 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-16T11:22:26.405 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-16T11:22:26.407 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-16T11:22:26.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:22:26.476 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:26.474+0000 7feb9de17440 -1 Falling back to public interface 2024-09-16T11:22:26.476 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:26.475+0000 7f8d86628440 -1 Falling back to public interface 2024-09-16T11:22:26.478 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:26.477+0000 7fedf820a440 -1 Falling back to public interface 2024-09-16T11:22:26.486 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:26.485+0000 7feb9de17440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:26.487 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:26.485+0000 7feb9de17440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:26.487 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:26.485+0000 7feb9de17440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:26.490 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:26.488+0000 7fedf820a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:26.490 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:26.489+0000 7fedf820a440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:26.490 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:26.489+0000 7fedf820a440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:26.493 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:26.492+0000 7feb9de17440 -1 osd.0 0 log_to_monitors true 2024-09-16T11:22:26.494 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:26.493+0000 7f8d86628440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-16T11:22:26.495 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:26.493+0000 7f8d86628440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:26.495 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:26.493+0000 7f8d86628440 -1 journal do_read_entry(8192): bad header magic 2024-09-16T11:22:26.497 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:26.496+0000 7fedf820a440 -1 osd.2 0 log_to_monitors true 2024-09-16T11:22:26.500 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:26.498+0000 7f8d86628440 -1 osd.1 0 log_to_monitors true 2024-09-16T11:22:26.578 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:26.578 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":5,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:22:26.388141+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:26.589 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-16T11:22:26.589 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T11:22:26.892 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:22:27.028 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.026+0000 7f9203dc4ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T11:22:27.039 INFO:teuthology.misc.health.smithi204.stdout: 2024-09-16T11:22:27.039 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":5,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:22:26.388141+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:27.048 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T11:22:27.072 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.070+0000 7f9203dc4ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T11:22:27.393 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.391+0000 7f9203dc4ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T11:22:27.476 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.474+0000 7f9203dc4ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T11:22:27.567 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.566+0000 7f9203dc4ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T11:22:27.610 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.608+0000 7f9203dc4ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T11:22:27.705 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.703+0000 7f9203dc4ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T11:22:27.757 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.756+0000 7f9203dc4ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T11:22:27.799 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.797+0000 7f9203dc4ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T11:22:27.895 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.893+0000 7f9203dc4ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T11:22:27.943 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.942+0000 7f9203dc4ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T11:22:27.992 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:27.990+0000 7f9203dc4ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T11:22:28.044 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:28.043+0000 7f9203dc4ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T11:22:28.086 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:28.085+0000 7f9203dc4ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T11:22:28.129 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:28.127+0000 7f9203dc4ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T11:22:28.395 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:28.393+0000 7f8d81df8640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T11:22:28.395 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:28.393+0000 7feb98df7640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T11:22:28.396 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:28.393+0000 7fedf31f7640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T11:22:28.397 INFO:tasks.ceph.cluster2.osd.0.smithi204.stderr:2024-09-16T11:22:28.395+0000 7feb955f0640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:22:28.397 INFO:tasks.ceph.cluster2.osd.2.smithi204.stderr:2024-09-16T11:22:28.395+0000 7fedef88c640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:22:28.397 INFO:tasks.ceph.cluster2.osd.1.smithi204.stderr:2024-09-16T11:22:28.395+0000 7f8d7ddf0640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T11:22:28.691 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:28.690+0000 7f9203dc4ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T11:22:29.703 INFO:tasks.ceph.cluster2.mgr.x.smithi053.stderr:2024-09-16T11:22:29.701+0000 7f91e8303640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T11:22:33.352 DEBUG:teuthology.orchestra.run.smithi204:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:22:33.607 INFO:teuthology.misc.health.smithi204.stdout: 2024-09-16T11:22:33.607 INFO:teuthology.misc.health.smithi204.stdout:{"epoch":12,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:22:33.405132+0000","last_up_change":"2024-09-16T11:22:29.390391+0000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:30.726822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":29226},{"type":"v1","addr":"172.21.15.204:6801","nonce":29226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":29226},{"type":"v1","addr":"172.21.15.204:6803","nonce":29226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":29226},{"type":"v1","addr":"172.21.15.204:6811","nonce":29226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":29226},{"type":"v1","addr":"172.21.15.204:6807","nonce":29226}]},"public_addr":"172.21.15.204:6801/29226","cluster_addr":"172.21.15.204:6803/29226","heartbeat_back_addr":"172.21.15.204:6811/29226","heartbeat_front_addr":"172.21.15.204:6807/29226","state":["exists","up"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":29227},{"type":"v1","addr":"172.21.15.204:6806","nonce":29227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":29227},{"type":"v1","addr":"172.21.15.204:6810","nonce":29227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":29227},{"type":"v1","addr":"172.21.15.204:6815","nonce":29227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":29227},{"type":"v1","addr":"172.21.15.204:6813","nonce":29227}]},"public_addr":"172.21.15.204:6806/29227","cluster_addr":"172.21.15.204:6810/29227","heartbeat_back_addr":"172.21.15.204:6815/29227","heartbeat_front_addr":"172.21.15.204:6813/29227","state":["exists","up"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":29228},{"type":"v1","addr":"172.21.15.204:6817","nonce":29228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":29228},{"type":"v1","addr":"172.21.15.204:6819","nonce":29228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":29228},{"type":"v1","addr":"172.21.15.204:6823","nonce":29228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":29228},{"type":"v1","addr":"172.21.15.204:6821","nonce":29228}]},"public_addr":"172.21.15.204:6817/29228","cluster_addr":"172.21.15.204:6819/29228","heartbeat_back_addr":"172.21.15.204:6823/29228","heartbeat_front_addr":"172.21.15.204:6821/29228","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.482114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.515043+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.540557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:33.619 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T11:22:33.619 INFO:tasks.ceph:Creating RBD pool 2024-09-16T11:22:33.619 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-16T11:22:34.620 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'rbd' created 2024-09-16T11:22:34.638 DEBUG:teuthology.orchestra.run.smithi204:> rbd --cluster cluster2 pool init rbd 2024-09-16T11:22:37.644 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-16T11:22:37.645 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-16T11:22:37.645 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-16T11:22:37.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-16T11:22:37.645 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T11:22:37.956 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:37.967 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":30068},{"type":"v1","addr":"172.21.15.53:6827","nonce":30068}]},"active_addr":"172.21.15.53:6827/30068","active_change":"2024-09-16T11:22:28.692154+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":3046222647}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2490599804}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":48742854}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":991186199}]}]} 2024-09-16T11:22:37.970 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-16T11:22:37.970 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-16T11:22:37.970 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:22:38.232 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:38.232 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":16,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:22:37.626380+0000","last_up_change":"2024-09-16T11:22:29.390391+0000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:30.726822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:33.891246+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":29226},{"type":"v1","addr":"172.21.15.204:6801","nonce":29226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":29226},{"type":"v1","addr":"172.21.15.204:6803","nonce":29226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":29226},{"type":"v1","addr":"172.21.15.204:6811","nonce":29226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":29226},{"type":"v1","addr":"172.21.15.204:6807","nonce":29226}]},"public_addr":"172.21.15.204:6801/29226","cluster_addr":"172.21.15.204:6803/29226","heartbeat_back_addr":"172.21.15.204:6811/29226","heartbeat_front_addr":"172.21.15.204:6807/29226","state":["exists","up"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":29227},{"type":"v1","addr":"172.21.15.204:6806","nonce":29227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":29227},{"type":"v1","addr":"172.21.15.204:6810","nonce":29227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":29227},{"type":"v1","addr":"172.21.15.204:6815","nonce":29227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":29227},{"type":"v1","addr":"172.21.15.204:6813","nonce":29227}]},"public_addr":"172.21.15.204:6806/29227","cluster_addr":"172.21.15.204:6810/29227","heartbeat_back_addr":"172.21.15.204:6815/29227","heartbeat_front_addr":"172.21.15.204:6813/29227","state":["exists","up"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":29228},{"type":"v1","addr":"172.21.15.204:6817","nonce":29228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":29228},{"type":"v1","addr":"172.21.15.204:6819","nonce":29228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":29228},{"type":"v1","addr":"172.21.15.204:6823","nonce":29228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":29228},{"type":"v1","addr":"172.21.15.204:6821","nonce":29228}]},"public_addr":"172.21.15.204:6817/29228","cluster_addr":"172.21.15.204:6819/29228","heartbeat_back_addr":"172.21.15.204:6823/29228","heartbeat_front_addr":"172.21.15.204:6821/29228","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.482114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.515043+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.540557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:38.245 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-16T11:22:38.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:22:38.504 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:38.505 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":16,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:22:37.626380+0000","last_up_change":"2024-09-16T11:22:29.390391+0000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:30.726822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:33.891246+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":29226},{"type":"v1","addr":"172.21.15.204:6801","nonce":29226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":29226},{"type":"v1","addr":"172.21.15.204:6803","nonce":29226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":29226},{"type":"v1","addr":"172.21.15.204:6811","nonce":29226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":29226},{"type":"v1","addr":"172.21.15.204:6807","nonce":29226}]},"public_addr":"172.21.15.204:6801/29226","cluster_addr":"172.21.15.204:6803/29226","heartbeat_back_addr":"172.21.15.204:6811/29226","heartbeat_front_addr":"172.21.15.204:6807/29226","state":["exists","up"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":29227},{"type":"v1","addr":"172.21.15.204:6806","nonce":29227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":29227},{"type":"v1","addr":"172.21.15.204:6810","nonce":29227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":29227},{"type":"v1","addr":"172.21.15.204:6815","nonce":29227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":29227},{"type":"v1","addr":"172.21.15.204:6813","nonce":29227}]},"public_addr":"172.21.15.204:6806/29227","cluster_addr":"172.21.15.204:6810/29227","heartbeat_back_addr":"172.21.15.204:6815/29227","heartbeat_front_addr":"172.21.15.204:6813/29227","state":["exists","up"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":29228},{"type":"v1","addr":"172.21.15.204:6817","nonce":29228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":29228},{"type":"v1","addr":"172.21.15.204:6819","nonce":29228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":29228},{"type":"v1","addr":"172.21.15.204:6823","nonce":29228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":29228},{"type":"v1","addr":"172.21.15.204:6821","nonce":29228}]},"public_addr":"172.21.15.204:6817/29228","cluster_addr":"172.21.15.204:6819/29228","heartbeat_back_addr":"172.21.15.204:6823/29228","heartbeat_front_addr":"172.21.15.204:6821/29228","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.482114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.515043+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.540557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:22:38.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-16T11:22:38.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-16T11:22:38.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-16T11:22:38.648 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2024-09-16T11:22:38.649 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T11:22:38.664 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2024-09-16T11:22:38.664 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T11:22:38.665 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2024-09-16T11:22:38.665 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T11:22:38.935 INFO:teuthology.orchestra.run.smithi204.stdout:34359738370 2024-09-16T11:22:38.948 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2024-09-16T11:22:38.982 INFO:teuthology.orchestra.run.smithi204.stdout:34359738370 2024-09-16T11:22:38.994 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.2 2024-09-16T11:22:38.999 INFO:teuthology.orchestra.run.smithi204.stdout:34359738370 2024-09-16T11:22:39.012 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.1 2024-09-16T11:22:39.950 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T11:22:39.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T11:22:40.013 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T11:22:40.253 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2024-09-16T11:22:40.266 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.0 2024-09-16T11:22:40.267 DEBUG:teuthology.parallel:result is None 2024-09-16T11:22:40.291 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2024-09-16T11:22:40.303 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.2 2024-09-16T11:22:40.303 DEBUG:teuthology.parallel:result is None 2024-09-16T11:22:40.320 INFO:teuthology.orchestra.run.smithi204.stdout:34359738371 2024-09-16T11:22:40.332 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.1 2024-09-16T11:22:40.332 DEBUG:teuthology.parallel:result is None 2024-09-16T11:22:40.332 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T11:22:40.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:22:40.630 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:40.630 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:22:40.642 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T11:22:40.579767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2389824,"kb_used_data":278649152,"kb_used_omap":298,"kb_used_meta":0,"kb_avail":278649024,"statfs":{"total":287783780352,"available":285336600576,"internally_reserved":131072,"allocated":285336731648,"data_stored":285336731648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":306079,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.958418"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638479+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:22:37.638479+0000","last_peered":"2024-09-16T11:22:37.638479+0000","last_clean":"2024-09-16T11:22:37.638479+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:22:37.638479+0000","last_undegraded":"2024-09-16T11:22:37.638479+0000","last_fullsized":"2024-09-16T11:22:37.638479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638317+0000","last_change":"2024-09-16T11:22:37.638530+0000","last_active":"2024-09-16T11:22:37.638317+0000","last_peered":"2024-09-16T11:22:37.638317+0000","last_clean":"2024-09-16T11:22:37.638317+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:22:37.638317+0000","last_undegraded":"2024-09-16T11:22:37.638317+0000","last_fullsized":"2024-09-16T11:22:37.638317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:18.738957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638583+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:22:37.638583+0000","last_peered":"2024-09-16T11:22:37.638583+0000","last_clean":"2024-09-16T11:22:37.638583+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:22:37.638583+0000","last_undegraded":"2024-09-16T11:22:37.638583+0000","last_fullsized":"2024-09-16T11:22:37.638583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:56.608460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638038+0000","last_change":"2024-09-16T11:22:37.638272+0000","last_active":"2024-09-16T11:22:37.638038+0000","last_peered":"2024-09-16T11:22:37.638038+0000","last_clean":"2024-09-16T11:22:37.638038+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:22:37.638038+0000","last_undegraded":"2024-09-16T11:22:37.638038+0000","last_fullsized":"2024-09-16T11:22:37.638038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:34.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.635551+0000","last_change":"2024-09-16T11:22:37.635551+0000","last_active":"2024-09-16T11:22:37.635551+0000","last_peered":"2024-09-16T11:22:37.635551+0000","last_clean":"2024-09-16T11:22:37.635551+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:22:37.635551+0000","last_undegraded":"2024-09-16T11:22:37.635551+0000","last_fullsized":"2024-09-16T11:22:37.635551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:04.960663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:38.486156+0000","last_change":"2024-09-16T11:22:38.486357+0000","last_active":"2024-09-16T11:22:38.486156+0000","last_peered":"2024-09-16T11:22:38.486156+0000","last_clean":"2024-09-16T11:22:38.486156+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:22:38.486156+0000","last_undegraded":"2024-09-16T11:22:38.486156+0000","last_fullsized":"2024-09-16T11:22:38.486156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:39.481341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:38.486188+0000","last_change":"2024-09-16T11:22:38.486359+0000","last_active":"2024-09-16T11:22:38.486188+0000","last_peered":"2024-09-16T11:22:38.486188+0000","last_clean":"2024-09-16T11:22:38.486188+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:22:38.486188+0000","last_undegraded":"2024-09-16T11:22:38.486188+0000","last_fullsized":"2024-09-16T11:22:38.486188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:54.166730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638191+0000","last_change":"2024-09-16T11:22:37.638406+0000","last_active":"2024-09-16T11:22:37.638191+0000","last_peered":"2024-09-16T11:22:37.638191+0000","last_clean":"2024-09-16T11:22:37.638191+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:22:37.638191+0000","last_undegraded":"2024-09-16T11:22:37.638191+0000","last_fullsized":"2024-09-16T11:22:37.638191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:42.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":81,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:36.632038+0000","last_change":"2024-09-16T11:22:32.471966+0000","last_active":"2024-09-16T11:22:36.632038+0000","last_peered":"2024-09-16T11:22:36.632038+0000","last_clean":"2024-09-16T11:22:36.632038+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:22:36.632038+0000","last_undegraded":"2024-09-16T11:22:36.632038+0000","last_fullsized":"2024-09-16T11:22:36.632038+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_clean_scrub_stamp":"2024-09-16T11:22:31.398024+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:25.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795764,"kb_used_data":92883880,"kb_used_omap":60,"kb_used_meta":0,"kb_avail":92883852,"statfs":{"total":95927926784,"available":95113064448,"internally_reserved":28672,"allocated":95113093120,"data_stored":95113093120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62362,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797036,"kb_used_data":92882636,"kb_used_omap":128,"kb_used_meta":0,"kb_avail":92882580,"statfs":{"total":95927926784,"available":95111761920,"internally_reserved":57344,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131211,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797024,"kb_used_data":92882636,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882592,"statfs":{"total":95927926784,"available":95111774208,"internally_reserved":45056,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112506,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]}],"pool_statfs":[]}} 2024-09-16T11:22:40.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:22:40.901 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:40.902 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:22:40.913 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T11:22:40.701168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2389824,"kb_used_data":278649152,"kb_used_omap":298,"kb_used_meta":0,"kb_avail":278649024,"statfs":{"total":287783780352,"available":285336600576,"internally_reserved":131072,"allocated":285336731648,"data_stored":285336731648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":306079,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079819"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638479+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:22:37.638479+0000","last_peered":"2024-09-16T11:22:37.638479+0000","last_clean":"2024-09-16T11:22:37.638479+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:22:37.638479+0000","last_undegraded":"2024-09-16T11:22:37.638479+0000","last_fullsized":"2024-09-16T11:22:37.638479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638317+0000","last_change":"2024-09-16T11:22:37.638530+0000","last_active":"2024-09-16T11:22:37.638317+0000","last_peered":"2024-09-16T11:22:37.638317+0000","last_clean":"2024-09-16T11:22:37.638317+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:22:37.638317+0000","last_undegraded":"2024-09-16T11:22:37.638317+0000","last_fullsized":"2024-09-16T11:22:37.638317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:18.738957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638583+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:22:37.638583+0000","last_peered":"2024-09-16T11:22:37.638583+0000","last_clean":"2024-09-16T11:22:37.638583+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:22:37.638583+0000","last_undegraded":"2024-09-16T11:22:37.638583+0000","last_fullsized":"2024-09-16T11:22:37.638583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:56.608460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638038+0000","last_change":"2024-09-16T11:22:37.638272+0000","last_active":"2024-09-16T11:22:37.638038+0000","last_peered":"2024-09-16T11:22:37.638038+0000","last_clean":"2024-09-16T11:22:37.638038+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:22:37.638038+0000","last_undegraded":"2024-09-16T11:22:37.638038+0000","last_fullsized":"2024-09-16T11:22:37.638038+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:34.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.635551+0000","last_change":"2024-09-16T11:22:37.635551+0000","last_active":"2024-09-16T11:22:37.635551+0000","last_peered":"2024-09-16T11:22:37.635551+0000","last_clean":"2024-09-16T11:22:37.635551+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:22:37.635551+0000","last_undegraded":"2024-09-16T11:22:37.635551+0000","last_fullsized":"2024-09-16T11:22:37.635551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:04.960663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:38.486156+0000","last_change":"2024-09-16T11:22:38.486357+0000","last_active":"2024-09-16T11:22:38.486156+0000","last_peered":"2024-09-16T11:22:38.486156+0000","last_clean":"2024-09-16T11:22:38.486156+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:22:38.486156+0000","last_undegraded":"2024-09-16T11:22:38.486156+0000","last_fullsized":"2024-09-16T11:22:38.486156+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:39.481341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:38.486188+0000","last_change":"2024-09-16T11:22:38.486359+0000","last_active":"2024-09-16T11:22:38.486188+0000","last_peered":"2024-09-16T11:22:38.486188+0000","last_clean":"2024-09-16T11:22:38.486188+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:22:38.486188+0000","last_undegraded":"2024-09-16T11:22:38.486188+0000","last_fullsized":"2024-09-16T11:22:38.486188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:54.166730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T11:22:37.638191+0000","last_change":"2024-09-16T11:22:37.638406+0000","last_active":"2024-09-16T11:22:37.638191+0000","last_peered":"2024-09-16T11:22:37.638191+0000","last_clean":"2024-09-16T11:22:37.638191+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:22:37.638191+0000","last_undegraded":"2024-09-16T11:22:37.638191+0000","last_fullsized":"2024-09-16T11:22:37.638191+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:42.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":81,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T11:22:36.632038+0000","last_change":"2024-09-16T11:22:32.471966+0000","last_active":"2024-09-16T11:22:36.632038+0000","last_peered":"2024-09-16T11:22:36.632038+0000","last_clean":"2024-09-16T11:22:36.632038+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:22:36.632038+0000","last_undegraded":"2024-09-16T11:22:36.632038+0000","last_fullsized":"2024-09-16T11:22:36.632038+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_clean_scrub_stamp":"2024-09-16T11:22:31.398024+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:25.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795764,"kb_used_data":92883880,"kb_used_omap":60,"kb_used_meta":0,"kb_avail":92883852,"statfs":{"total":95927926784,"available":95113064448,"internally_reserved":28672,"allocated":95113093120,"data_stored":95113093120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62362,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797036,"kb_used_data":92882636,"kb_used_omap":128,"kb_used_meta":0,"kb_avail":92882580,"statfs":{"total":95927926784,"available":95111761920,"internally_reserved":57344,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":131211,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797024,"kb_used_data":92882636,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882592,"statfs":{"total":95927926784,"available":95111774208,"internally_reserved":45056,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112506,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]}],"pool_statfs":[]}} 2024-09-16T11:22:40.914 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T11:22:40.914 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-16T11:22:40.914 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-16T11:22:40.914 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-16T11:22:41.204 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:22:41.204 INFO:teuthology.orchestra.run.smithi204.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T11:22:41.216 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-16T11:22:41.216 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T11:22:41.226 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T11:22:41.226 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:22:41.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:41.557 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.4 2024-09-16T11:22:41.570 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:41.899 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.5 2024-09-16T11:22:41.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:42.239 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.6 2024-09-16T11:22:42.252 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:42.580 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.4 2024-09-16T11:22:42.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:42.919 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.5 2024-09-16T11:22:42.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:43.259 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.6 2024-09-16T11:22:43.272 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T11:22:43.281 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T11:22:43.281 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:22:43.281 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:43.613 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror 2024-09-16T11:22:43.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:43.955 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.0 2024-09-16T11:22:43.968 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:44.292 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.1 2024-09-16T11:22:44.305 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:44.633 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.2 2024-09-16T11:22:44.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:44.976 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.3 2024-09-16T11:22:44.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:45.317 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror 2024-09-16T11:22:45.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:45.664 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.0 2024-09-16T11:22:45.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:46.010 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.1 2024-09-16T11:22:46.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:46.355 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.2 2024-09-16T11:22:46.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T11:22:46.701 INFO:teuthology.orchestra.run.smithi204.stderr:updated caps for client.mirror.3 2024-09-16T11:22:46.714 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T11:22:46.728 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:22:46.729 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T11:22:46.729 DEBUG:teuthology.orchestra.run.smithi204:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:22:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:22:46.745 INFO:teuthology.orchestra.run.smithi204.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-16T11:22:46.745 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:22:46.801 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:22:46.801 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-16T11:22:46.857 INFO:tasks.workunit:timeout=3h 2024-09-16T11:22:46.857 INFO:tasks.workunit:cleanup=True 2024-09-16T11:22:46.857 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-16T11:22:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-16T11:23:37.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T11:23:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: git switch -c 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Or undo this operation with: 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: git switch - 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T11:23:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: 2024-09-16T11:23:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T11:23:37.248 DEBUG:teuthology.orchestra.run.smithi204:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-16T11:23:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T11:23:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T11:23:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T11:23:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T11:23:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T11:23:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T11:23:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T11:23:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T11:23:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T11:23:38.190 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-16T11:23:38.191 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-16T11:23:38.207 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_journal.sh on cluster1.client.mirror... 2024-09-16T11:23:38.209 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_journal.sh... 2024-09-16T11:23:38.209 DEBUG:teuthology.orchestra.run.smithi204:workunit test rbd/rbd_mirror_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2024-09-16T11:23:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ type xmlstarlet 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ XMLSTARLET=xmlstarlet 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ RBD_MIRROR_INSTANCES=2 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CLUSTER1=cluster1 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CLUSTER2=cluster2 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ POOL=mirror 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ PARENT_POOL=mirror_parent 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ NS1=ns1 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ NS2=ns2 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ TEMPDIR= 2024-09-16T11:23:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ID=mirror 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_POOL_MODE=pool 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ MIRROR_IMAGE_MODE=journal 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS='--id mirror' 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ LAST_MIRROR_INSTANCE=1 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_BIN=. 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_SRC=. 2024-09-16T11:23:38.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 -gt 0 ']' 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local c 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_tempdir 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mktemp -d 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ TEMPDIR=/tmp/tmp.MThcdnqr7J 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z 1 ']' 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_pools cluster1 cluster2 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_map_file 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_addr 2024-09-16T11:23:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local admin_key_file 2024-09-16T11:23:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid 2024-09-16T11:23:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-16T11:23:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' already exists 2024-09-16T11:23:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-16T11:23:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' already exists 2024-09-16T11:23:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-16T11:23:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-16T11:23:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-09-16T11:23:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-16T11:23:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-16T11:23:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-16T11:23:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-09-16T11:23:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 mirror cluster2 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:23:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-16T11:23:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=e42af5e2-7604-45cd-a805-83c0884d2aeb 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:23:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-16T11:23:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:23:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2024-09-16T11:23:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-09-16T11:23:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-09-16T11:23:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-09-16T11:23:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-09-16T11:23:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:23:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:23:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-16T11:23:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=bfb4ade8-4ad6-4796-946b-bc8a2d66727d 2024-09-16T11:23:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:23:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:23:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:23:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ setup_pools cluster2 cluster1 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_map_file 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mon_addr 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local admin_key_file 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:46.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-16T11:23:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' already exists 2024-09-16T11:23:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-16T11:23:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' already exists 2024-09-16T11:23:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-16T11:23:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-16T11:23:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-09-16T11:23:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-16T11:23:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-16T11:23:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-16T11:23:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-09-16T11:23:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-16T11:23:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-09-16T11:23:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z ']' 2024-09-16T11:23:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 mirror cluster1 2024-09-16T11:23:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-09-16T11:23:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-09-16T11:23:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:23:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:23:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=114e3311-4d8a-4a56-99da-6d6d8fa992e8 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:23:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-16T11:23:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=e00bde97-9ea2-42af-8304-8ce7048b6ce4 2024-09-16T11:23:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:23:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:23:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:23:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-09-16T11:23:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:23:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n octopus ']' 2024-09-16T11:23:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-09-16T11:23:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:set require_min_compat_client to octopus 2024-09-16T11:23:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:23:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-09-16T11:23:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:set require_min_compat_client to octopus 2024-09-16T11:23:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: add image and test replay' 2024-09-16T11:23:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:23:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:23:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:23:57 TEST: add image and test replay 2024-09-16T11:23:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:23:57 TEST: add image and test replay 2024-09-16T11:23:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-09-16T11:23:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:23:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-09-16T11:23:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:23:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:23:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:23:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:23:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:23:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:23:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:23:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:23:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=test 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:23:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:23:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:23:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:23:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2024-09-16T11:23:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror test key1 value1 2024-09-16T11:23:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:23:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:23:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key1 2024-09-16T11:23:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=value1 2024-09-16T11:23:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key1 value1 2024-09-16T11:23:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror test key2 value2 2024-09-16T11:23:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:23:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:23:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key2 2024-09-16T11:23:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=value2 2024-09-16T11:23:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key2 value2 2024-09-16T11:23:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:23:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:23:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:23:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:23:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:23:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:23:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:23:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:23:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:23:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:23:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:23:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:23:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:23:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:23:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:23:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:23:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:23:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:23:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:23:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:23:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:23:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:23:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:23:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:23:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:23:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:23:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:23:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:23:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:23:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:23:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:23:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:23:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:23:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:23:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:23:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:23:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:23:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:23:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:23:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:23:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:23:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:23:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:23:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:23:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:23:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:23:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:23:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:23:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:23:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:23:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:23:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:23:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:23:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:23:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:23:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:23:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:23:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:23:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:23:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:23:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:23:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:23:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:23:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:23:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:23:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:23:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:23:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:23:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:24:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:00.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:24:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:24:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:24:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:24:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:24:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:24:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:24:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:24:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:24:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:24:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:24:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:24:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:24:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3125.05 bytes/sec: 12 MiB/s 2024-09-16T11:24:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:24:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:24:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:24:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:24:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:24:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:24:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:24:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:24:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=1' '!=' '' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=1' = '' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:24:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=4 2024-09-16T11:24:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-09-16T11:24:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:24:03.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=3 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=1' 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 4 = 3 -a 101 -le 1 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:24:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:24:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:03.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:24:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:24:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:24:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=4, entry_tid=101' -a 'object_number=1, tag_tid=4, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:24:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:24:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-09-16T11:24:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:01' 2024-09-16T11:24:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:24:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-09-16T11:24:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:01' 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:01' 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:01' 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142534259mirror.0smithi20417.2.7-1643-gd75f2382trueOK42604263mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}42534259mirror.0smithi2042024-09-16 11:24:01' 2024-09-16T11:24:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:01' 2024-09-16T11:24:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test down+unknown 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test down+unknown '' '' 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:24:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:01' 2024-09-16T11:24:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*down+unknown' 2024-09-16T11:24:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: down+unknown 2024-09-16T11:24:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: status not found 2024-09-16T11:24:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-09-16T11:24:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: ' 2024-09-16T11:24:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test down+unknown '' 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local prev_last_update= 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='UNKNOWNUNKNOWNOK1test0bf12ffc-5d8b-49fe-90f8-30f79104b611down+unknownstatus not foundcluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}2024-09-16 11:24:01' 2024-09-16T11:24:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-09-16T11:24:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=down+unknown 2024-09-16T11:24:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-09-16T11:24:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep down+unknown 2024-09-16T11:24:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:down+unknown 2024-09-16T11:24:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:24:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:24:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:24:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:24:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:24:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:24:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:24:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:24:05.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_meta cluster1 mirror test key1 value1 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key1 2024-09-16T11:24:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local value=value1 2024-09-16T11:24:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key1 2024-09-16T11:24:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test value1 = value1 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_image_meta cluster1 mirror test key2 value2 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=key2 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local value=value2 2024-09-16T11:24:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 image-meta get mirror/test key2 2024-09-16T11:24:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test value2 = value2 2024-09-16T11:24:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: stop mirror, add image, start mirror and test replay' 2024-09-16T11:24:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:24:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:24:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:24:06 TEST: stop mirror, add image, start mirror and test replay 2024-09-16T11:24:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:24:06 TEST: stop mirror, add image, start mirror and test replay 2024-09-16T11:24:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-09-16T11:24:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:24:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T11:24:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:24:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:24:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:24:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:24:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:24:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=31732 2024-09-16T11:24:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 31732 ']' 2024-09-16T11:24:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 31732 2024-09-16T11:24:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:24:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:24:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=31732 '$2 == pid {print; exit 1}' 2024-09-16T11:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:ubuntu 31732 2.5 0.1 543960 51840 ? Ssl 11:23 0:00 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:24:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:24:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:24:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=31732 '$2 == pid {print; exit 1}' 2024-09-16T11:24:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:24:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:24:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=31732 '$2 == pid {print; exit 1}' 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-09-16T11:24:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:24:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:24:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-09-16T11:24:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:24:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:24:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:24:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:24:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:24:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:24:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=31760 2024-09-16T11:24:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 31760 ']' 2024-09-16T11:24:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 31760 2024-09-16T11:24:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:24:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:24:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=31760 '$2 == pid {print; exit 1}' 2024-09-16T11:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=31760 '$2 == pid {print; exit 1}' 2024-09-16T11:24:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-16T11:24:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:24:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:24:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image1=test1 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test1 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:24:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:10.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test1 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test1 100 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:24:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test1 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:24:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:24:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:24:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2631.62 bytes/sec: 10 MiB/s 2024-09-16T11:24:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-09-16T11:24:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:24:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:24:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:24:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-09-16T11:24:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:24:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2024-09-16T11:24:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:24:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:24:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2024-09-16T11:24:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-09-16T11:24:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:24:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:24:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-09-16T11:24:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test1 2024-09-16T11:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:24:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2024-09-16T11:24:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test1'] 2024-09-16T11:24:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:24:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:24:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:24:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test1 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test1 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:24:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test1 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test1 ']' 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test1' 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test1 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test1 2024-09-16T11:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test1 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test1 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test1 '' 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:15.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-09-16T11:24:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:24:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test1.status 2024-09-16T11:24:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test1.status 2024-09-16T11:24:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.status 2024-09-16T11:24:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1 2024-09-16T11:24:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.status 2024-09-16T11:24:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:24:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:24:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.status 2024-09-16T11:24:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:24:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test1 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test1 '..*' 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test1 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test1.status 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test1.status 2024-09-16T11:24:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.status 2024-09-16T11:24:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test1 2024-09-16T11:24:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.status 2024-09-16T11:24:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:24:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:24:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.status 2024-09-16T11:24:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:24:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:24:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position 2024-09-16T11:24:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:24:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position '' 2024-09-16T11:24:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:24:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:14' 2024-09-16T11:24:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:24:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:16.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:14' 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-09-16 11:24:14' 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:14' 2024-09-16T11:24:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}42884295mirror.0smithi2042024-09-16 11:24:14test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:14' 2024-09-16T11:24:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:14' 2024-09-16T11:24:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:24:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test1 down+unknown 2024-09-16T11:24:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test1 down+unknown '' '' 2024-09-16T11:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-09-16T11:24:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test1 2024-09-16T11:24:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:24:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:14' 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*down+unknown' 2024-09-16T11:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: down+unknown 2024-09-16T11:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: status not found 2024-09-16T11:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-09-16T11:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: ' 2024-09-16T11:24:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-09-16T11:24:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test1 down+unknown '' 2024-09-16T11:24:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=down+unknown 2024-09-16T11:24:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local prev_last_update= 2024-09-16T11:24:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='UNKNOWNUNKNOWNOK2test0bf12ffc-5d8b-49fe-90f8-30f79104b611down+unknownstatus not foundcluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}2024-09-16 11:24:14test10d3e4142-a2d8-4071-afab-a36d9f0bed19down+unknownstatus not foundcluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2024-09-16 11:24:14' 2024-09-16T11:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update= 2024-09-16T11:24:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=down+unknown 2024-09-16T11:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo down+unknown 2024-09-16T11:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep down+unknown 2024-09-16T11:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:down+unknown 2024-09-16T11:24:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test1 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test1.export 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test1.export 2024-09-16T11:24:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.export 2024-09-16T11:24:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test1.export 2024-09-16T11:24:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test1.export 2024-09-16T11:24:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test1.export 2024-09-16T11:24:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test1.export 2024-09-16T11:24:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test1 /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.export 2024-09-16T11:24:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test1 /tmp/tmp.MThcdnqr7J/cluster1-mirror-test1.export 2024-09-16T11:24:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test1.export 2024-09-16T11:24:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test1.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test1.export 2024-09-16T11:24:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: test the first image is replaying after restart' 2024-09-16T11:24:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:24:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:24:18 TEST: test the first image is replaying after restart 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:24:18 TEST: test the first image is replaying after restart 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:24:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:24:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:24:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:24:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:24:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3030.35 bytes/sec: 12 MiB/s 2024-09-16T11:24:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:24:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:24:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:24:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:24:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:24:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2024-09-16T11:24:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=101' '!=' '' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=4, entry_tid=101' = '' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:24:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=5 2024-09-16T11:24:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:24:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=4 2024-09-16T11:24:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:24:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:24:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:24:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=4, entry_tid=101' 2024-09-16T11:24:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:24:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:24:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=101 2024-09-16T11:24:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 5 = 4 -a 101 -le 101 2024-09-16T11:24:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:24:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:24:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:24:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:24:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:24:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:24:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:24:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:24:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:24:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:24:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:24:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:24:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:24:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:24:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2024-09-16T11:24:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:14' 2024-09-16T11:24:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:24:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2024-09-16T11:24:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:14' 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:14' 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:14' 2024-09-16T11:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}}42884295mirror.0smithi2042024-09-16 11:24:14test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:14' 2024-09-16T11:24:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:14' 2024-09-16T11:24:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:24:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:24:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:24:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:24:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:24:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:24:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: stop/start/restart mirror via admin socket' 2024-09-16T11:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:24:21 TEST: stop/start/restart mirror via admin socket 2024-09-16T11:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:24:21 TEST: stop/start/restart mirror via admin socket 2024-09-16T11:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2024-09-16T11:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:24:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:24:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:24:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:24:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:24:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:24:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:24:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:24:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:24:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:21' 2024-09-16T11:24:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:24:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:24:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:24:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:21' 2024-09-16T11:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:24:21' 2024-09-16T11:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:21' 2024-09-16T11:24:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:21test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:21' 2024-09-16T11:24:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:21' 2024-09-16T11:24:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:24:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:24:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:24:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:24:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:21' 2024-09-16T11:24:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:24:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:24:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:24:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:21' 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-09-16 11:24:21' 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:21' 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:21test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:21' 2024-09-16T11:24:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:21' 2024-09-16T11:24:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:24:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:24:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror start 2024-09-16T11:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start 2024-09-16T11:24:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:28.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:26' 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:26' 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:26' 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:26' 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:26test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:26' 2024-09-16T11:24:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:26' 2024-09-16T11:24:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:26' 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:26' 2024-09-16T11:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-09-16 11:24:26' 2024-09-16T11:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:30.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:26' 2024-09-16T11:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:26test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:26' 2024-09-16T11:24:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:26' 2024-09-16T11:24:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2024-09-16T11:24:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:33.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:24:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:31' 2024-09-16T11:24:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:31' 2024-09-16T11:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:31' 2024-09-16T11:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:31' 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:31test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:31' 2024-09-16T11:24:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:31' 2024-09-16T11:24:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-09-16T11:24:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-09-16T11:24:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:31' 2024-09-16T11:24:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:31' 2024-09-16T11:24:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-09-16 11:24:31' 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:31' 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:31test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:31' 2024-09-16T11:24:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:31' 2024-09-16T11:24:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2024-09-16T11:24:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2024-09-16T11:24:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2024-09-16T11:24:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2024-09-16T11:24:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:24:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:24:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:24:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:24:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:24:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:24:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:24:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:24:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:35' 2024-09-16T11:24:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:24:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:24:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:24:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:35' 2024-09-16T11:24:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:24:35' 2024-09-16T11:24:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:35' 2024-09-16T11:24:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:35test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:35' 2024-09-16T11:24:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:35' 2024-09-16T11:24:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:24:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:24:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:39.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-09-16T11:24:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:24:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:24:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:35' 2024-09-16T11:24:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:24:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:24:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:24:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:35' 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-09-16 11:24:35' 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:35' 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:35test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:35' 2024-09-16T11:24:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:35' 2024-09-16T11:24:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:24:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:24:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2024-09-16T11:24:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2024-09-16T11:24:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:40.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:24:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:24:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:40' 2024-09-16T11:24:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:40' 2024-09-16T11:24:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:40' 2024-09-16T11:24:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:40' 2024-09-16T11:24:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:40test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:40' 2024-09-16T11:24:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:40' 2024-09-16T11:24:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-09-16T11:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:40' 2024-09-16T11:24:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:40' 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-09-16 11:24:40' 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:40' 2024-09-16T11:24:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:40test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:40' 2024-09-16T11:24:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:40' 2024-09-16T11:24:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2024-09-16T11:24:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2024-09-16T11:24:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2024-09-16T11:24:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:46.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:24:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:24:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:24:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:24:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:24:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:24:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:24:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:44' 2024-09-16T11:24:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:24:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:24:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:24:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:44' 2024-09-16T11:24:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:24:44' 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:44' 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:44test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:44' 2024-09-16T11:24:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:44' 2024-09-16T11:24:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:24:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-09-16T11:24:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:44' 2024-09-16T11:24:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:24:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:24:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:24:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:44' 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-09-16 11:24:44' 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:44' 2024-09-16T11:24:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:44test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:44' 2024-09-16T11:24:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:44' 2024-09-16T11:24:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:24:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:24:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:24:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:24:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2024-09-16T11:24:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:49' 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:49' 2024-09-16T11:24:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:49' 2024-09-16T11:24:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:49' 2024-09-16T11:24:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:49test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:44' 2024-09-16T11:24:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:49' 2024-09-16T11:24:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror start mirror cluster2 2024-09-16T11:24:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror cluster2 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:51.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror cluster2 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror cluster2 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror cluster2 2024-09-16T11:24:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:24:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:51' 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:24:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:51' 2024-09-16T11:24:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-09-16 11:24:51' 2024-09-16T11:24:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:51' 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:49test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:51' 2024-09-16T11:24:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:24:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:51' 2024-09-16T11:24:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:24:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test 2024-09-16T11:24:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:24:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:24:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:24:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:24:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:24:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:54' 2024-09-16T11:24:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:24:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:24:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:24:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:24:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:24:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:24:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:24:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:24:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:24:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:24:56.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:54' 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:24:54' 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:54' 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:24:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:24:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:24:54test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:24:51' 2024-09-16T11:24:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:24:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:24:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:54' 2024-09-16T11:24:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:24:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:24:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:24:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:24:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:24:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2024-09-16T11:24:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:24:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:24:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:24:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:24:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:24:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:24:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:24:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:24:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:24:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:24:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:24:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:24:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:24:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:24:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:24:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:24:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:24:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:24:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:24:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:24:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:24:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:24:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:24:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2024-09-16T11:24:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:24:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:24:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:24:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:25:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:25:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:25:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2024-09-16T11:25:01.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:25:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:25:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:25:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:25:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:59' 2024-09-16T11:25:02.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:25:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:25:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:25:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:25:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:25:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:59' 2024-09-16T11:25:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:24:59' 2024-09-16T11:25:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:59' 2024-09-16T11:25:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:59test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:59' 2024-09-16T11:25:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:59' 2024-09-16T11:25:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:25:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:25:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:25:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:25:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped '' '' 2024-09-16T11:25:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:24:59' 2024-09-16T11:25:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:25:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:25:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: stopped 2024-09-16T11:25:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:25:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:25:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:59' 2024-09-16T11:25:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped '2024-09-16 11:24:59' 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:24:59' 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:59test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+stoppedstopped42884295mirror.0smithi2042024-09-16 11:24:59' 2024-09-16T11:25:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:25:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:24:59' 2024-09-16T11:25:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:25:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:25:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:25:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:25:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:25:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2024-09-16T11:25:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:25:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2024-09-16T11:25:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:25:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2024-09-16T11:25:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test1 started 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test1 2024-09-16T11:25:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test1 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:05.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2024-09-16T11:25:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:25:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:03' 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:25:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2024-09-16T11:25:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:03' 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:25:03' 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:03' 2024-09-16T11:25:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:25:03test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:25:03' 2024-09-16T11:25:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:03' 2024-09-16T11:25:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:25:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:07.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:25:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2024-09-16T11:25:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying '' '' 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test1 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test1: 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0d3e4142-a2d8-4071-afab-a36d9f0bed19 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:03' 2024-09-16T11:25:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:25:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:25:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:03' 2024-09-16T11:25:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying '2024-09-16 11:25:03' 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:03' 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:25:03test10d3e4142-a2d8-4071-afab-a36d9f0bed19up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:25:03' 2024-09-16T11:25:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/last_update' 2024-09-16T11:25:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:03' 2024-09-16T11:25:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test1'\'']/state' 2024-09-16T11:25:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:25:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:25:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool= 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image= 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush 2024-09-16T11:25:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ all_admin_daemons cluster1 rbd mirror status 2024-09-16T11:25:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror status 2024-09-16T11:25:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2024-09-16T11:25:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-09-16T11:25:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool_replayers": [ 2024-09-16T11:25:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: e42af5e2-7604-45cd-a805-83c0884d2aeb cluster: cluster2 client: client.mirror", 2024-09-16T11:25:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror", 2024-09-16T11:25:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4295", 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4295", 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": true, 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instances": [ 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4295", 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4297" 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.32525.cluster1.94878796240936.asok", 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.32525.cluster2.94878808512552.asok", 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [ 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "mirror/test", 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "mirror/test1", 2024-09-16T11:25:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [ 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns1", 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-09-16T11:25:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns2", 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-09-16T11:25:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: bfb4ade8-4ad6-4796-946b-bc8a2d66727d cluster: cluster2 client: client.mirror", 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror_parent", 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4300", 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4300", 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": true, 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instances": [ 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4300", 2024-09-16T11:25:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "4303" 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ], 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.32525.cluster1.94878821980200.asok", 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.32525.cluster2.94878834333736.asok", 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter": { 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_deleter_status": { 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "delete_images_queue": [], 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "failed_deletes_queue": [] 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [] 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-09-16T11:25:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror status 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:25:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:08.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool_replayers": [ 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: e42af5e2-7604-45cd-a805-83c0884d2aeb cluster: cluster2 client: client.mirror", 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror", 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4297", 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-09-16T11:25:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4295", 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": false, 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.32553.cluster1.94623101810728.asok", 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.32553.cluster2.94623114000424.asok", 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [ 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns1", 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [] 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "ns2", 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [] 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: { 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "peer": "uuid: bfb4ade8-4ad6-4796-946b-bc8a2d66727d cluster: cluster2 client: client.mirror", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "pool": "mirror_parent", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "instance_id": "4303", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "stopped (manual)", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader_instance_id": "4300", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "leader": false, 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.32553.cluster1.94623127435304.asok", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.32553.cluster2.94623139616808.asok", 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "sync_throttler": { 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 5, 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "deletion_throttler": { 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "max_parallel_requests": 1, 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "running_requests": 0, 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "waiting_requests": 0 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: }, 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "image_replayers": [], 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "namespaces": [] 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: } 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: ] 2024-09-16T11:25:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-09-16T11:25:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test1 2024-09-16T11:25:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:25:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test1 2024-09-16T11:25:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test1 2024-09-16T11:25:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test1 2024-09-16T11:25:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test1 2024-09-16T11:25:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:25:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: test image rename' 2024-09-16T11:25:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:25:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:25:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:25:08 TEST: test image rename 2024-09-16T11:25:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:25:08 TEST: test image rename 2024-09-16T11:25:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ new_name=test_RENAMED 2024-09-16T11:25:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_image cluster2 mirror test test_RENAMED 2024-09-16T11:25:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_name=test_RENAMED 2024-09-16T11:25:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rename mirror/test mirror/test_RENAMED 2024-09-16T11:25:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2024-09-16T11:25:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2024-09-16T11:25:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2024-09-16T11:25:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2024-09-16T11:25:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_RENAMED: 2024-09-16T11:25:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:25:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2024-09-16T11:25:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:10' 2024-09-16T11:25:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:25:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2024-09-16T11:25:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:10' 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2024-09-16 11:25:10' 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:10' 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest_RENAMED0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}}42884295mirror.0smithi2042024-09-16 11:25:10' 2024-09-16T11:25:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2024-09-16T11:25:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:10' 2024-09-16T11:25:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2024-09-16T11:25:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:25:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:25:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:10.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-09-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "mirror/test_RENAMED", 2024-09-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test_RENAMED 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test_RENAMED 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test_RENAMED 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying '' '' 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2024-09-16T11:25:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_RENAMED: 2024-09-16T11:25:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0bf12ffc-5d8b-49fe-90f8-30f79104b611 2024-09-16T11:25:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2024-09-16T11:25:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:11' 2024-09-16T11:25:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:25:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2024-09-16T11:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:11' 2024-09-16T11:25:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying '2024-09-16 11:25:11' 2024-09-16T11:25:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:11' 2024-09-16T11:25:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest_RENAMED0bf12ffc-5d8b-49fe-90f8-30f79104b611up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}}42884295mirror.0smithi2042024-09-16 11:25:11' 2024-09-16T11:25:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/last_update' 2024-09-16T11:25:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:11' 2024-09-16T11:25:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_RENAMED'\'']/state' 2024-09-16T11:25:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:25:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:25:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_image cluster2 mirror test_RENAMED test 2024-09-16T11:25:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_RENAMED 2024-09-16T11:25:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_name=test 2024-09-16T11:25:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rename mirror/test_RENAMED mirror/test 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:14.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:no valid command found; 1 closest matches: 2024-09-16T11:25:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status mirror/test_RENAMED 2024-09-16T11:25:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:25:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:25:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: test trash move restore' 2024-09-16T11:25:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:25:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:25:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:25:15 TEST: test trash move restore 2024-09-16T11:25:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:25:15 TEST: test trash move restore 2024-09-16T11:25:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster2 mirror test 2024-09-16T11:25:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster2 info mirror/test 2024-09-16T11:25:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:25:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10a14eae05e 2024-09-16T11:25:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trash_move cluster2 mirror test 2024-09-16T11:25:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 trash move mirror/test 2024-09-16T11:25:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-16T11:25:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:25:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:25:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:25:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:25:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:25:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:25:15.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:25:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:25:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ trash_restore cluster2 mirror 10a14eae05e 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10a14eae05e 2024-09-16T11:25:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 trash restore mirror/10a14eae05e 2024-09-16T11:25:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:25:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:25:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:25:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:25:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:18.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:25:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:25:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:25:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:25:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed (with rbd-mirror on one cluster)' 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:25:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:25:20 TEST: check if removed 'images'\''' OMAP are removed '(with' rbd-mirror on one 'cluster)' 2024-09-16T11:25:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:25:20 TEST: check if removed images' OMAP are removed (with rbd-mirror on one cluster) 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-16T11:25:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-16T11:25:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:25:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-16T11:25:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-16T11:25:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:25:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-09-16T11:25:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:25:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-09-16T11:25:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:25:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:25:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-09-16T11:25:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:25:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:25:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:25:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:25:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:25:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:25:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-09-16T11:25:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-09-16T11:25:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-09-16T11:25:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:25:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:25:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-09-16T11:25:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-09-16T11:25:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-09-16T11:25:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:25:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:25:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:25:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-09-16T11:25:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:25:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-16T11:25:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:25:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-09-16T11:25:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:25:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-09-16T11:25:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:25:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:25:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:25:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:25:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:25:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error getting omap key set mirror/rbd_mirror_leader: (2) No such file or directory 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image_and_enable_mirror cluster2 mirror test 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=journal 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:25:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' pool = image ']' 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' mirror = mirror_parent ']' 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:25:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:25:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:25:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:25:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:25:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:25:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:25:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:25:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:25:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:25:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:25:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:25:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:25:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:25:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:25:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:25:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:25:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2941.22 bytes/sec: 11 MiB/s 2024-09-16T11:25:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:25:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:25:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:25:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:25:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:25:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:25:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:25:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:25:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:28.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:25:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-09-16T11:25:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:25:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-09-16T11:25:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:25:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:25:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:25:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-09-16T11:25:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 1 -a 101 -le 3 2024-09-16T11:25:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:25:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:25:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:25:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:25:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:29.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=5], [object_number=0, tag_tid=2, entry_tid=4], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-16T11:25:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:25:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=5], [object_number=0, tag_tid=2, entry_tid=4], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-16T11:25:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=5' 2024-09-16T11:25:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=5' 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=5' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=5' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=5' 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-09-16T11:25:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=5' 2024-09-16T11:25:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:25:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=2 2024-09-16T11:25:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:25:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:25:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=5' 2024-09-16T11:25:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:25:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=5 2024-09-16T11:25:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 -a 101 -le 5 2024-09-16T11:25:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:25:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:25:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:29.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:29.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:30.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:30.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:25:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:27' 2024-09-16T11:25:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:25:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:25:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:30.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:27' 2024-09-16T11:25:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:25:27' 2024-09-16T11:25:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:27' 2024-09-16T11:25:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42884295mirror.0smithi2042024-09-16 11:25:27' 2024-09-16T11:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:27' 2024-09-16T11:25:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:25:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:30.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:25:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: failover and failback' 2024-09-16T11:25:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:25:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:25:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:25:30 TEST: failover and failback 2024-09-16T11:25:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:25:30 TEST: failover and failback 2024-09-16T11:25:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster2 2024-09-16T11:25:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:0 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:30.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:25:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:25:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:25:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:1 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:25:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:25:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:25:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:25:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:25:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:25:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:25:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:25:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:25:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:25:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:25:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:25:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:25:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:32 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:33' 2024-09-16T11:25:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:25:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:25:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:25:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:35 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:35' 2024-09-16T11:25:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:25:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:25:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:35' 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:25:35' 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:35' 2024-09-16T11:25:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:25:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:25:35' 2024-09-16T11:25:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:35' 2024-09-16T11:25:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:25:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:25:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:25:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:35 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:35' 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:25:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:25:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:35' 2024-09-16T11:25:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:25:35' 2024-09-16T11:25:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:35' 2024-09-16T11:25:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:25:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:25:35' 2024-09-16T11:25:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:35' 2024-09-16T11:25:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:25:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:25:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:25:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:25:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:25:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:25:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:25:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:25:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:25:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:41.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:25:41.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:25:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:25:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:25:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2777.82 bytes/sec: 11 MiB/s 2024-09-16T11:25:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:25:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:25:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:25:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:25:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:25:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:25:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:25:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:42.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:25:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' '' 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' = '' 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=6 2024-09-16T11:25:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:25:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=5 2024-09-16T11:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:25:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:25:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:25:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:25:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:25:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 6 = 5 -a 101 -le 0 2024-09-16T11:25:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:25:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:25:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:25:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:25:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=5, entry_tid=0], [object_number=1, tag_tid=2, entry_tid=101], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:25:42.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=5, entry_tid=0' '!=' 'object_number=0, tag_tid=5, entry_tid=0' 2024-09-16T11:25:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:25:42.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:25:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:25:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:25:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:25:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:25:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:25:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:25:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:25:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:25:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=6, entry_tid=101], [object_number=0, tag_tid=6, entry_tid=100], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:25:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:25:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=6, entry_tid=101' -a 'object_number=1, tag_tid=6, entry_tid=101' = 'object_number=1, tag_tid=6, entry_tid=101' 2024-09-16T11:25:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:25:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:25:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:40 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2024-09-16T11:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:40' 2024-09-16T11:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:25:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:25:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:25:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:40' 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:25:40' 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:40' 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:25:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}2024-09-16 11:25:40' 2024-09-16T11:25:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:40' 2024-09-16T11:25:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:25:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:25:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:25:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:25:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:25:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:25:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:25:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2024-09-16T11:25:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:40 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:40' 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:25:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:25:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}} 2024-09-16T11:25:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:40' 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:25:40' 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:40' 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":4},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":5}}42884295mirror.0smithi2042024-09-16 11:25:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:25:40' 2024-09-16T11:25:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:40' 2024-09-16T11:25:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:25:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:25:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:25:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:25:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:25:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:25:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:25:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:25:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:25:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:25:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:25:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:25:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:25:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:25:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:25:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:25:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:25:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:25:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:25:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:25:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:25:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:25:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:25:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:25:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:25:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:25:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:25:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:25:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:48.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:46 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:45' 2024-09-16T11:25:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:25:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:25:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:25:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:46 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:45' 2024-09-16T11:25:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:25:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:25:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:25:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:50 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:50' 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:25:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:25:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:50' 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:25:50' 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:50' 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:25:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:25:50' 2024-09-16T11:25:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:50' 2024-09-16T11:25:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:25:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:25:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:25:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:25:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:50 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:50' 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:25:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:25:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:25:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:25:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:25:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:25:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:25:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:50' 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:25:50' 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:25:50' 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:25:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:25:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:25:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:25:50' 2024-09-16T11:25:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:25:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:25:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:25:50' 2024-09-16T11:25:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:25:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:25:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:25:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:25:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:25:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:25:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:25:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:25:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:25:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:25:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:25:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:25:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:25:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:25:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:25:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:25:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:25:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:25:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:25:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:25:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:25:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:25:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:25:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:25:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:25:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:25:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:25:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:25:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:25:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:25:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:25:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:25:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:25:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:25:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:25:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:25:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:25:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:25:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:25:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:25:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:25:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:25:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:25:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:25:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:25:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:25:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:25:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:25:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:25:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:25:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:25:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:25:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:25:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:00 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:25:58' 2024-09-16T11:26:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:00' 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:26:00' 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:00' 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:26:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:25:58' 2024-09-16T11:26:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:00' 2024-09-16T11:26:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:26:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:00 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:00' 2024-09-16T11:26:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:00' 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:26:00' 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:00' 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:26:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:26:00' 2024-09-16T11:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:00' 2024-09-16T11:26:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:26:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:26:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:26:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:26:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:03.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:26:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:05.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:26:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:09.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:26:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:26:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=11, entry_tid=0' 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=11, entry_tid=0' 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=11, entry_tid=0' -a 'object_number=0, tag_tid=11, entry_tid=0' = 'object_number=0, tag_tid=11, entry_tid=0' 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:10 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:10' 2024-09-16T11:26:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:26:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:26:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:26:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2024-09-16T11:26:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:26:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:26:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:10' 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:26:10' 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:10' 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}42884295mirror.0smithi2042024-09-16 11:26:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:26:10' 2024-09-16T11:26:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:10' 2024-09-16T11:26:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:26:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:26:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:26:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:10 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:10' 2024-09-16T11:26:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:26:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:26:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:26:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:10' 2024-09-16T11:26:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:26:10' 2024-09-16T11:26:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:10' 2024-09-16T11:26:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:26:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}}2024-09-16 11:26:10' 2024-09-16T11:26:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:10' 2024-09-16T11:26:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:26:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:26:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:26:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:26:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:26:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:26:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:26:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:26:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:26:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:26:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:26:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:26:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:26:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:26:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:26:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:26:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:26:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:26:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:15 2024-09-16T11:26:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:15' 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:15' 2024-09-16T11:26:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:26:15' 2024-09-16T11:26:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:15' 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:26:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:26:15' 2024-09-16T11:26:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:15' 2024-09-16T11:26:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:15 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:15' 2024-09-16T11:26:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:15' 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:26:15' 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:15' 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:26:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:26:15' 2024-09-16T11:26:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:15' 2024-09-16T11:26:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:26:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:26:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:26:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:26:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:26:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:26:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:26:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:26:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:26:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:26:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:26:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:26:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:26:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster1 mirror test 100 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:26:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:26:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:26:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:26:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:26:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:26:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:26:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3846.21 bytes/sec: 15 MiB/s 2024-09-16T11:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:26:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:26:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:26:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:26:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:26:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:26:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:26:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:26:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:26:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-09-16T11:26:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-09-16T11:26:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos= 2024-09-16T11:26:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = '' 2024-09-16T11:26:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test '' '!=' '' 2024-09-16T11:26:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-09-16T11:26:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:26:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:26:22.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:26:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:26:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:26:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-09-16T11:26:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=12, entry_tid=0]]], state=connected] 2024-09-16T11:26:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos= 2024-09-16T11:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = '' 2024-09-16T11:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test '' '!=' '' 2024-09-16T11:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:26:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:26:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-09-16T11:26:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:26:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=1, tag_tid=13, entry_tid=101], [object_number=0, tag_tid=13, entry_tid=100], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=13, entry_tid=101' 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=13, entry_tid=101' -a 'object_number=1, tag_tid=13, entry_tid=101' = 'object_number=1, tag_tid=13, entry_tid=101' 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:20 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:20' 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:26:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:26:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:20' 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:26:20' 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:20' 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:26:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}2024-09-16 11:26:20' 2024-09-16T11:26:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:20' 2024-09-16T11:26:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:26:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:26:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying primary_position 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying primary_position '' 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:20 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:25' 2024-09-16T11:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:26:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:26:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:26:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}} 2024-09-16T11:26:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:26:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:26:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:20' 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:26:20' 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:20' 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":9},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":12}}44464452mirror.0smithi2042024-09-16 11:26:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:26:25' 2024-09-16T11:26:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:20' 2024-09-16T11:26:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:26:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:26:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:26:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:26:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:26:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:26:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:26:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:26:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:26:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:26:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:26:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:26:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:26:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:26:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:25 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:26:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:27' 2024-09-16T11:26:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:26:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:25 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:27' 2024-09-16T11:26:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:26:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:30 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:30' 2024-09-16T11:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:30' 2024-09-16T11:26:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:26:30' 2024-09-16T11:26:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:30' 2024-09-16T11:26:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:26:30cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:26:30' 2024-09-16T11:26:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:30' 2024-09-16T11:26:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:30 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:30' 2024-09-16T11:26:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:30' 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:26:30' 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:30' 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:26:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:26:35' 2024-09-16T11:26:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:30' 2024-09-16T11:26:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:26:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:26:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:26:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:26:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:26:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:26:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test 100 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:26:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:26:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:26:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:26:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:26:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:26:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2857.18 bytes/sec: 11 MiB/s 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:26:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:26:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:26:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:26:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:26:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:26:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:26:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:43.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' '' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' = '' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:26:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:26:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=18 2024-09-16T11:26:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:26:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:26:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=17 2024-09-16T11:26:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:26:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:26:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:26:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:26:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:26:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:26:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 18 = 17 -a 101 -le 0 2024-09-16T11:26:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:26:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:26:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:26:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:26:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:26:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:26:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-09-16T11:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:26:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:26:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:26:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=6, entry_tid=101], [object_number=3, tag_tid=6, entry_tid=99], [object_number=2, tag_tid=6, entry_tid=98]]], state=connected] 2024-09-16T11:26:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=17, entry_tid=0' '!=' 'object_number=0, tag_tid=17, entry_tid=0' 2024-09-16T11:26:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.8 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:26:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:45.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:45.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:26:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:26:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:26:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:26:45.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:26:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:26:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=18, entry_tid=101], [object_number=0, tag_tid=18, entry_tid=100], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:26:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=18, entry_tid=101' -a 'object_number=1, tag_tid=18, entry_tid=101' = 'object_number=1, tag_tid=18, entry_tid=101' 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:26:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:40 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:45' 2024-09-16T11:26:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:26:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:26:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:26:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2024-09-16T11:26:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:26:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:26:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:40' 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:26:40' 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:40' 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}42884295mirror.0smithi2042024-09-16 11:26:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:26:45' 2024-09-16T11:26:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:40' 2024-09-16T11:26:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:26:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:26:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:26:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:26:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:45 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:40' 2024-09-16T11:26:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:26:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:26:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:26:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:45' 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:26:45' 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:45' 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:26:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":13},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}}2024-09-16 11:26:40' 2024-09-16T11:26:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:45' 2024-09-16T11:26:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:26:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:26:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:26:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:26:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:26:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:26:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:26:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:26:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:26:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: failover / failback loop' 2024-09-16T11:26:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:26:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:26:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:26:48 TEST: failover / failback loop 2024-09-16T11:26:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:26:48 TEST: failover / failback loop 2024-09-16T11:26:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 20 2024-09-16T11:26:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:26:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:26:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:26:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:26:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:26:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:26:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:26:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:26:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:26:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:26:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:26:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:26:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:26:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:26:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:26:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:50 2024-09-16T11:26:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:50' 2024-09-16T11:26:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:50' 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:26:50' 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:50' 2024-09-16T11:26:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:26:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:26:50' 2024-09-16T11:26:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:50' 2024-09-16T11:26:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:50 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:50' 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:26:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:26:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:50' 2024-09-16T11:26:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:26:50' 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:50' 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:26:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:26:50' 2024-09-16T11:26:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:50' 2024-09-16T11:26:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:26:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:26:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:26:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:26:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:26:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:26:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:26:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:26:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:26:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:26:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:26:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:26:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:26:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:26:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:26:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:26:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:26:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:26:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:26:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:26:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:26:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:26:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:26:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:26:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:26:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:26:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:26:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:26:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:26:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:26:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:26:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:26:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:26:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:26:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:26:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:26:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:26:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:26:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:26:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:26:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:26:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:26:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:26:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:26:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=19, entry_tid=0' 2024-09-16T11:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:26:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:26:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:26:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:26:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:26:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:26:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=19, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:26:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=19, entry_tid=0' 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=19, entry_tid=0' -a 'object_number=0, tag_tid=19, entry_tid=0' = 'object_number=0, tag_tid=19, entry_tid=0' 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:26:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:26:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:26:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:26:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:55 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:55' 2024-09-16T11:26:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:26:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:26:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:26:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:26:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:26:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:26:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:55' 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:26:55' 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:55' 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:26:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:26:55cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}2024-09-16 11:26:55' 2024-09-16T11:26:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:26:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:26:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:55' 2024-09-16T11:26:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:26:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:26:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:26:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:26:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:26:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:26:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:55 2024-09-16T11:27:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:26:55' 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:27:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}} 2024-09-16T11:27:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:55' 2024-09-16T11:27:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:26:55' 2024-09-16T11:27:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:26:55' 2024-09-16T11:27:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":15},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19}}44464452mirror.0smithi2042024-09-16 11:26:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:26:55' 2024-09-16T11:27:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:26:55' 2024-09-16T11:27:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:27:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:27:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:27:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:27:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:27:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:27:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:03.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:03.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:27:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:00 2024-09-16T11:27:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:01' 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:00' 2024-09-16T11:27:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:27:00' 2024-09-16T11:27:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:00' 2024-09-16T11:27:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:27:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:27:01' 2024-09-16T11:27:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:00' 2024-09-16T11:27:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:01 2024-09-16T11:27:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:00' 2024-09-16T11:27:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:27:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:05 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:05' 2024-09-16T11:27:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:05' 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:27:05' 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:05' 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:27:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:27:05' 2024-09-16T11:27:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:07.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:07.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:05' 2024-09-16T11:27:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:27:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:27:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:27:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:27:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:27:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:27:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:27:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:27:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:27:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:27:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:27:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:27:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:27:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:27:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:27:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:11.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:27:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:27:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:27:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:27:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=23, entry_tid=0' 2024-09-16T11:27:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:27:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:27:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=23, entry_tid=0' 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=23, entry_tid=0' -a 'object_number=0, tag_tid=23, entry_tid=0' = 'object_number=0, tag_tid=23, entry_tid=0' 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:10 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:10' 2024-09-16T11:27:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:27:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:27:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:27:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:10' 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:27:10' 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:10' 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:27:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}2024-09-16 11:27:10' 2024-09-16T11:27:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:10' 2024-09-16T11:27:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:27:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:27:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:27:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:27:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2024-09-16T11:27:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:10 2024-09-16T11:27:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:10' 2024-09-16T11:27:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:27:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:27:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2024-09-16T11:27:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:10' 2024-09-16T11:27:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:27:10' 2024-09-16T11:27:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:10' 2024-09-16T11:27:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":20},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}}42884295mirror.0smithi2042024-09-16 11:27:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:27:10' 2024-09-16T11:27:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:10' 2024-09-16T11:27:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:27:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:27:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:27:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:27:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:27:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:27:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:14.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:27:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:27:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:15 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:15' 2024-09-16T11:27:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:15' 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:27:15' 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:15' 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:27:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:27:15' 2024-09-16T11:27:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:15' 2024-09-16T11:27:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:15 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:15' 2024-09-16T11:27:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:19.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:15' 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:27:15' 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:15' 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:27:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:27:15' 2024-09-16T11:27:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:15' 2024-09-16T11:27:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:27:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:27:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:27:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:27:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:27:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:27:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:27:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:27:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:27:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:27:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:27:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:27:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:27:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:27:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:27:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:27:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=24, entry_tid=0' 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:27:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:27:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=24, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=24, entry_tid=0' 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=24, entry_tid=0' -a 'object_number=0, tag_tid=24, entry_tid=0' = 'object_number=0, tag_tid=24, entry_tid=0' 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:20 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:20' 2024-09-16T11:27:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:27:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:27:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:27:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:20' 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:27:20' 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:20' 2024-09-16T11:27:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:27:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}2024-09-16 11:27:20' 2024-09-16T11:27:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:20' 2024-09-16T11:27:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:27:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:27:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:20 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:20' 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:27:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:27:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}} 2024-09-16T11:27:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:20' 2024-09-16T11:27:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:27:20' 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:20' 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":21},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":24}}44464452mirror.0smithi2042024-09-16 11:27:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:27:20' 2024-09-16T11:27:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:20' 2024-09-16T11:27:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:27:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:27:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:27:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:27:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:25 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:26' 2024-09-16T11:27:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:25' 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:27:25' 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:25' 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:27:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:27:26' 2024-09-16T11:27:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:25' 2024-09-16T11:27:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:26 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:25' 2024-09-16T11:27:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:27:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:26 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:25' 2024-09-16T11:27:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:27:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:30 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:30' 2024-09-16T11:27:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:30' 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:27:30' 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:30' 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:27:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:27:30' 2024-09-16T11:27:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:30' 2024-09-16T11:27:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:27:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:27:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:27:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:27:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:35.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:27:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:27:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:27:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:27:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:27:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:27:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:27:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:27:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:27:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:27:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:27:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=28, entry_tid=0' 2024-09-16T11:27:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:27:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=28, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:27:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=28, entry_tid=0' 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=28, entry_tid=0' -a 'object_number=0, tag_tid=28, entry_tid=0' = 'object_number=0, tag_tid=28, entry_tid=0' 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:27:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:35 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2024-09-16T11:27:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:35' 2024-09-16T11:27:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:27:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:27:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:27:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:35' 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:27:35' 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:35' 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:27:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}2024-09-16 11:27:35' 2024-09-16T11:27:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:35' 2024-09-16T11:27:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:27:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:27:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:27:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:27:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:38.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:35 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:35' 2024-09-16T11:27:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:27:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:27:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}} 2024-09-16T11:27:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:35' 2024-09-16T11:27:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:27:35' 2024-09-16T11:27:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:35' 2024-09-16T11:27:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":25},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":28}}42884295mirror.0smithi2042024-09-16 11:27:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:27:35' 2024-09-16T11:27:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:35' 2024-09-16T11:27:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:27:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:27:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:27:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:27:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:27:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:27:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:27:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:27:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:27:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:27:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:40 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:40' 2024-09-16T11:27:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:40' 2024-09-16T11:27:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:27:40' 2024-09-16T11:27:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:40' 2024-09-16T11:27:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:27:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:27:40' 2024-09-16T11:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:40' 2024-09-16T11:27:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:40 2024-09-16T11:27:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:40' 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:40' 2024-09-16T11:27:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:27:40' 2024-09-16T11:27:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:43.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:40' 2024-09-16T11:27:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:27:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:27:40' 2024-09-16T11:27:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:40' 2024-09-16T11:27:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:27:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:27:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:27:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:27:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:27:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:27:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:27:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:27:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:27:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:27:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:27:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=29, entry_tid=0' 2024-09-16T11:27:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:27:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:27:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:27:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:27:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:27:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:27:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:27:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=29, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:27:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:27:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=29, entry_tid=0' 2024-09-16T11:27:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=29, entry_tid=0' -a 'object_number=0, tag_tid=29, entry_tid=0' = 'object_number=0, tag_tid=29, entry_tid=0' 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:45 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:45' 2024-09-16T11:27:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:27:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:27:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:27:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:45' 2024-09-16T11:27:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:27:45' 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:45' 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:27:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}2024-09-16 11:27:45' 2024-09-16T11:27:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:45' 2024-09-16T11:27:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:27:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:27:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:27:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:27:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:27:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:27:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:45 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:45' 2024-09-16T11:27:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:27:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:27:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}} 2024-09-16T11:27:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:45' 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:27:45' 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:27:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:45' 2024-09-16T11:27:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":26},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":29}}44464452mirror.0smithi2042024-09-16 11:27:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:27:45' 2024-09-16T11:27:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:45' 2024-09-16T11:27:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:27:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:27:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:27:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:27:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:27:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:27:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:27:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:27:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:27:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:27:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:27:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:27:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:27:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:27:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:27:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:27:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:27:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:27:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:27:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:27:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:50 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:51' 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:50' 2024-09-16T11:27:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:27:50' 2024-09-16T11:27:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:50' 2024-09-16T11:27:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:27:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:27:51' 2024-09-16T11:27:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:50' 2024-09-16T11:27:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:27:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:51 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:50' 2024-09-16T11:27:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:27:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:27:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:27:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:55 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:27:55' 2024-09-16T11:27:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:27:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:27:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:27:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:27:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:27:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:27:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:27:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:27:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:27:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:27:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:55' 2024-09-16T11:27:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:27:55' 2024-09-16T11:27:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:27:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:27:55' 2024-09-16T11:27:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:27:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:27:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:27:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:27:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:27:55' 2024-09-16T11:27:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:27:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:27:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:27:55' 2024-09-16T11:27:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:27:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:27:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:27:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:27:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:27:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:27:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:27:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:27:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:27:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:27:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:27:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:27:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:27:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:27:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:27:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:27:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:27:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:27:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:27:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:27:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:27:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:27:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:27:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:27:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:27:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:27:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:27:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:27:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:27:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:27:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:27:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:27:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:27:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:27:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:27:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:27:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:27:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:27:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:27:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:00.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:28:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:28:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:28:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:28:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=33, entry_tid=0' 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=33, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:01.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=33, entry_tid=0' 2024-09-16T11:28:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=33, entry_tid=0' -a 'object_number=0, tag_tid=33, entry_tid=0' = 'object_number=0, tag_tid=33, entry_tid=0' 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:00 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:00' 2024-09-16T11:28:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:28:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:28:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:28:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:00' 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:28:00' 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:00' 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:28:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}2024-09-16 11:28:00' 2024-09-16T11:28:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:00' 2024-09-16T11:28:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:28:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:28:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:00 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:00' 2024-09-16T11:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:28:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}} 2024-09-16T11:28:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:00' 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:28:00' 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:00' 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":30},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":33}}42884295mirror.0smithi2042024-09-16 11:28:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:28:00' 2024-09-16T11:28:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:00' 2024-09-16T11:28:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:28:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:28:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:28:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:28:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:28:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:28:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:28:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:05 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:05' 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:05' 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:28:05' 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:05' 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:28:05cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:28:05' 2024-09-16T11:28:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:05' 2024-09-16T11:28:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:28:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:05 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:05' 2024-09-16T11:28:08.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:08.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:05' 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:28:05' 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:05' 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:28:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:28:05' 2024-09-16T11:28:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:05' 2024-09-16T11:28:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:28:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:28:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:28:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:28:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:28:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:09.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:28:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:28:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:28:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:12.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:28:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:28:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:28:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:28:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:28:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:28:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:28:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:28:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:28:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:28:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:28:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:28:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=34, entry_tid=0' 2024-09-16T11:28:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:28:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:28:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=34, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=34, entry_tid=0' 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=34, entry_tid=0' -a 'object_number=0, tag_tid=34, entry_tid=0' = 'object_number=0, tag_tid=34, entry_tid=0' 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:28:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:10 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:10' 2024-09-16T11:28:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:28:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:28:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:28:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:10' 2024-09-16T11:28:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:28:10' 2024-09-16T11:28:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:10' 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:28:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}2024-09-16 11:28:10' 2024-09-16T11:28:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:10' 2024-09-16T11:28:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:28:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:28:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:28:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2024-09-16T11:28:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:10 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:10' 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:28:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}} 2024-09-16T11:28:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:10' 2024-09-16T11:28:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:28:10' 2024-09-16T11:28:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:10' 2024-09-16T11:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":31},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":34}}44464452mirror.0smithi2042024-09-16 11:28:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:28:10' 2024-09-16T11:28:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:10' 2024-09-16T11:28:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:28:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:28:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:28:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:28:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:28:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:28:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:28:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:28:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:28:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:15 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:28:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:16' 2024-09-16T11:28:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:15' 2024-09-16T11:28:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:28:15' 2024-09-16T11:28:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:15' 2024-09-16T11:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:28:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:28:16' 2024-09-16T11:28:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:15' 2024-09-16T11:28:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:28:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:16 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:15' 2024-09-16T11:28:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:28:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:20 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:20' 2024-09-16T11:28:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:20' 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:28:20' 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:20' 2024-09-16T11:28:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:28:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:28:20' 2024-09-16T11:28:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:20' 2024-09-16T11:28:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:28:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:28:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:28:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:28:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:23.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:28:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:28:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:25.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:28:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:28:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:28:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:28:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:28:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:28:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:28:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:28:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:28:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:28:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:28:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:28:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:28:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:28:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=38, entry_tid=0' 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:28:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=38, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=38, entry_tid=0' 2024-09-16T11:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=38, entry_tid=0' -a 'object_number=0, tag_tid=38, entry_tid=0' = 'object_number=0, tag_tid=38, entry_tid=0' 2024-09-16T11:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:30 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:25' 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:28:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:28:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:28:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:30' 2024-09-16T11:28:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:28:30' 2024-09-16T11:28:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:30' 2024-09-16T11:28:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:28:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}2024-09-16 11:28:25' 2024-09-16T11:28:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:30' 2024-09-16T11:28:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:28:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:28:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:28:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:28:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:25 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:30' 2024-09-16T11:28:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:28:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:28:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}} 2024-09-16T11:28:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:25' 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:28:25' 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:25' 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":35},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":38}}42884295mirror.0smithi2042024-09-16 11:28:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:28:30' 2024-09-16T11:28:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:25' 2024-09-16T11:28:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:28:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:28:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:28:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:28:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:28:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:28:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:28:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:28:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:35.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:28:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:35 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:35' 2024-09-16T11:28:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:35' 2024-09-16T11:28:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:28:35' 2024-09-16T11:28:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:35' 2024-09-16T11:28:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:28:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:28:35' 2024-09-16T11:28:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:35' 2024-09-16T11:28:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:28:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:28:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:35 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:35' 2024-09-16T11:28:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:35' 2024-09-16T11:28:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:28:35' 2024-09-16T11:28:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:37.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:35' 2024-09-16T11:28:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:37.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:28:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:28:35' 2024-09-16T11:28:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:35' 2024-09-16T11:28:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:28:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:28:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:28:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:28:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:28:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:28:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:28:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:28:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:28:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:28:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:28:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:28:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:28:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=39, entry_tid=0' 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:28:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=39, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:28:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=39, entry_tid=0' 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=39, entry_tid=0' -a 'object_number=0, tag_tid=39, entry_tid=0' = 'object_number=0, tag_tid=39, entry_tid=0' 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:28:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:40 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2024-09-16T11:28:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:40' 2024-09-16T11:28:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:28:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:28:42.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:28:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:40' 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:28:40' 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:40' 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:28:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}2024-09-16 11:28:40' 2024-09-16T11:28:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:40' 2024-09-16T11:28:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:28:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:28:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:28:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:28:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:40 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:40' 2024-09-16T11:28:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:28:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:28:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}} 2024-09-16T11:28:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:40' 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:28:40' 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:40' 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":36},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":39}}44464452mirror.0smithi2042024-09-16 11:28:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:28:40' 2024-09-16T11:28:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:40' 2024-09-16T11:28:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:28:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:28:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:28:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:28:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:45.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:28:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:28:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:28:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:28:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:28:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:28:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:45 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:45' 2024-09-16T11:28:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:45' 2024-09-16T11:28:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:28:45' 2024-09-16T11:28:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:45' 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:28:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:28:45' 2024-09-16T11:28:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:45' 2024-09-16T11:28:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:48.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:45 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:45' 2024-09-16T11:28:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:45' 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:28:45' 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:45' 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:28:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:28:45' 2024-09-16T11:28:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:45' 2024-09-16T11:28:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:28:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:28:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:28:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:28:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:28:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:28:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:28:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:28:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:28:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=40, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=43, entry_tid=0' 2024-09-16T11:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=40, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=40, entry_tid=0' 2024-09-16T11:28:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=43, entry_tid=0' -a 'object_number=0, tag_tid=43, entry_tid=0' = 'object_number=0, tag_tid=40, entry_tid=0' 2024-09-16T11:28:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=40, entry_tid=0' '!=' '' 2024-09-16T11:28:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:28:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=40, entry_tid=0' = '' 2024-09-16T11:28:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=40, entry_tid=0' 2024-09-16T11:28:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=43, entry_tid=0' 2024-09-16T11:28:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:28:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:28:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=43 2024-09-16T11:28:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=40, entry_tid=0' 2024-09-16T11:28:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:28:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:28:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=40 2024-09-16T11:28:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=43, entry_tid=0' 2024-09-16T11:28:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:28:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:28:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-09-16T11:28:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=40, entry_tid=0' 2024-09-16T11:28:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:28:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:28:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:28:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 43 = 40 -a 0 -le 0 2024-09-16T11:28:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:28:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:28:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:28:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:28:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:28:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:28:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=43, entry_tid=0' 2024-09-16T11:28:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:28:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:28:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:28:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:28:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:28:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:28:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:28:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=43, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:28:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=43, entry_tid=0' 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=43, entry_tid=0' -a 'object_number=0, tag_tid=43, entry_tid=0' = 'object_number=0, tag_tid=43, entry_tid=0' 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:50 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:50' 2024-09-16T11:28:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:28:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:28:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:28:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:50' 2024-09-16T11:28:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:28:50' 2024-09-16T11:28:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:50' 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:28:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}2024-09-16 11:28:50' 2024-09-16T11:28:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:50' 2024-09-16T11:28:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:28:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:28:52.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:28:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:28:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:50 2024-09-16T11:28:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:50' 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:28:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}} 2024-09-16T11:28:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:50' 2024-09-16T11:28:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:28:50' 2024-09-16T11:28:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:50' 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":40},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":43}}42884295mirror.0smithi2042024-09-16 11:28:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:28:50' 2024-09-16T11:28:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:50' 2024-09-16T11:28:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:28:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:28:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:28:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:28:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:28:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:28:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:28:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:28:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:28:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:28:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:28:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:28:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:28:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:28:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:28:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:28:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:28:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:28:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:28:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:28:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:28:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:28:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:28:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:28:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:28:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:28:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:28:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:28:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:28:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:28:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:28:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:28:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:55 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:55' 2024-09-16T11:28:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:55' 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:28:55' 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:55' 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:28:55cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:28:55' 2024-09-16T11:28:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:55' 2024-09-16T11:28:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:28:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:28:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:28:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:28:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:28:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:28:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:28:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:55 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:28:55' 2024-09-16T11:28:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:28:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:28:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:28:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:28:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:28:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:28:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:28:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:28:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:28:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:28:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:55' 2024-09-16T11:28:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:28:55' 2024-09-16T11:28:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:28:55' 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:28:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:28:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:28:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:28:55' 2024-09-16T11:28:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:28:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:28:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:28:55' 2024-09-16T11:28:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:28:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:28:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:28:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:28:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:28:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:28:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:28:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:28:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:28:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:29:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:00.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:29:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:29:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:02.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:29:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:29:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:29:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:29:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:29:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:29:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:29:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:29:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:29:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:29:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=44, entry_tid=0' 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:29:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=44, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=44, entry_tid=0' 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=44, entry_tid=0' -a 'object_number=0, tag_tid=44, entry_tid=0' = 'object_number=0, tag_tid=44, entry_tid=0' 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:00 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:00' 2024-09-16T11:29:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:29:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:29:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:29:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:00' 2024-09-16T11:29:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:29:00' 2024-09-16T11:29:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:00' 2024-09-16T11:29:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:29:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}2024-09-16 11:29:00' 2024-09-16T11:29:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:00' 2024-09-16T11:29:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:04.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:29:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:29:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:04.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:00 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:00' 2024-09-16T11:29:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:29:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:29:05.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}} 2024-09-16T11:29:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:00' 2024-09-16T11:29:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:29:00' 2024-09-16T11:29:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:00' 2024-09-16T11:29:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":41},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":44}}44464452mirror.0smithi2042024-09-16 11:29:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:29:00' 2024-09-16T11:29:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:00' 2024-09-16T11:29:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:29:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:29:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:29:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:29:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:06.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:29:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:29:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:05 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:06' 2024-09-16T11:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:05' 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:29:05' 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:05' 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:29:05cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:29:06' 2024-09-16T11:29:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:05' 2024-09-16T11:29:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:06 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:05' 2024-09-16T11:29:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:29:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:06 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:05' 2024-09-16T11:29:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:29:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:14.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:10 2024-09-16T11:29:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:10' 2024-09-16T11:29:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:10' 2024-09-16T11:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:29:10' 2024-09-16T11:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:10' 2024-09-16T11:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:29:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:29:10' 2024-09-16T11:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:10' 2024-09-16T11:29:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:29:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:29:14.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:29:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:29:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:29:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:29:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:29:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:29:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:29:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:29:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:29:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:29:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:29:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=48, entry_tid=0' 2024-09-16T11:29:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:29:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:29:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:29:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=48, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=48, entry_tid=0' 2024-09-16T11:29:16.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=48, entry_tid=0' -a 'object_number=0, tag_tid=48, entry_tid=0' = 'object_number=0, tag_tid=48, entry_tid=0' 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:15 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:15' 2024-09-16T11:29:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:29:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:29:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:29:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:15' 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:29:15' 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:15' 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:29:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}2024-09-16 11:29:15' 2024-09-16T11:29:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:15' 2024-09-16T11:29:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:29:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:29:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:29:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:15 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:15' 2024-09-16T11:29:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:29:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:29:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}} 2024-09-16T11:29:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:15' 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:29:15' 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:15' 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":45},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":48}}42884295mirror.0smithi2042024-09-16 11:29:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:29:15' 2024-09-16T11:29:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:15' 2024-09-16T11:29:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:29:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:29:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:29:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:29:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:29:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:29:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:29:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:29:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:20 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:20' 2024-09-16T11:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:20' 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:29:20' 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:20' 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:29:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:29:20' 2024-09-16T11:29:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:20' 2024-09-16T11:29:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:20 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:20' 2024-09-16T11:29:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:20' 2024-09-16T11:29:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:29:20' 2024-09-16T11:29:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:20' 2024-09-16T11:29:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:29:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:29:20' 2024-09-16T11:29:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:20' 2024-09-16T11:29:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:29:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:29:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:29:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:27.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:29:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:29:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:29:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:29:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:29:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:29:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:29:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=49, entry_tid=0' 2024-09-16T11:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:29:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=49, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=49, entry_tid=0' 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=49, entry_tid=0' -a 'object_number=0, tag_tid=49, entry_tid=0' = 'object_number=0, tag_tid=49, entry_tid=0' 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:25 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:25' 2024-09-16T11:29:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:29:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:29:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:29:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:25' 2024-09-16T11:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:29:25' 2024-09-16T11:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:28.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:25' 2024-09-16T11:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:29:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}2024-09-16 11:29:25' 2024-09-16T11:29:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:25' 2024-09-16T11:29:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:29:28.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:29:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:25 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:25' 2024-09-16T11:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}} 2024-09-16T11:29:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:25' 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:29:25' 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:25' 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":46},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":49}}44464452mirror.0smithi2042024-09-16 11:29:25cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:29:25' 2024-09-16T11:29:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:25' 2024-09-16T11:29:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:29:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:29:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:29:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:29:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:29:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:30 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:29:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:31' 2024-09-16T11:29:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:30' 2024-09-16T11:29:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:29:30' 2024-09-16T11:29:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:30' 2024-09-16T11:29:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:34.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:29:30cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:29:31' 2024-09-16T11:29:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:30' 2024-09-16T11:29:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:31 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:30' 2024-09-16T11:29:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:29:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:35 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:35' 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:35' 2024-09-16T11:29:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:29:35' 2024-09-16T11:29:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:35' 2024-09-16T11:29:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:29:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:29:35' 2024-09-16T11:29:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:35' 2024-09-16T11:29:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:29:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:29:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:29:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:29:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:29:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:29:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:29:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:29:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:29:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:29:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:29:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:29:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:29:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:29:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:29:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:29:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=53, entry_tid=0' 2024-09-16T11:29:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:29:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:29:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=53, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:29:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=53, entry_tid=0' 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=53, entry_tid=0' -a 'object_number=0, tag_tid=53, entry_tid=0' = 'object_number=0, tag_tid=53, entry_tid=0' 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:29:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:40 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:40' 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:29:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:29:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:29:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:40' 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:29:40' 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:40' 2024-09-16T11:29:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:29:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}2024-09-16 11:29:40' 2024-09-16T11:29:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:40' 2024-09-16T11:29:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:29:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:40 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:40' 2024-09-16T11:29:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:29:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:29:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}} 2024-09-16T11:29:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:40' 2024-09-16T11:29:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:29:40' 2024-09-16T11:29:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:40' 2024-09-16T11:29:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:43.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":50},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":53}}42884295mirror.0smithi2042024-09-16 11:29:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:29:40' 2024-09-16T11:29:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:40' 2024-09-16T11:29:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:29:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:29:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:29:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:29:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:29:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:29:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:29:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:29:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:29:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:29:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:29:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:29:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:29:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:29:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:29:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:29:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:29:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:29:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:29:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:29:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:45 2024-09-16T11:29:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:45' 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:45' 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:29:45' 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:45' 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:29:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:29:45' 2024-09-16T11:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:45' 2024-09-16T11:29:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:29:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:45 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:45' 2024-09-16T11:29:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:45' 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:29:45' 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:45' 2024-09-16T11:29:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:29:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:29:45' 2024-09-16T11:29:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:45' 2024-09-16T11:29:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:29:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:29:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:29:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:50.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:29:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:29:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:29:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:29:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:29:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:29:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:29:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:29:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:29:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:29:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:29:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=54, entry_tid=0' 2024-09-16T11:29:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:29:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:29:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:29:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:29:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=54, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:29:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=54, entry_tid=0' 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=54, entry_tid=0' -a 'object_number=0, tag_tid=54, entry_tid=0' = 'object_number=0, tag_tid=54, entry_tid=0' 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:50 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2024-09-16T11:29:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:50' 2024-09-16T11:29:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:29:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:29:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:29:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:50' 2024-09-16T11:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:29:50' 2024-09-16T11:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:29:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:50' 2024-09-16T11:29:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:29:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}2024-09-16 11:29:50' 2024-09-16T11:29:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:50' 2024-09-16T11:29:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:29:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:29:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:29:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:50 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:29:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:50' 2024-09-16T11:29:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:29:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:29:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}} 2024-09-16T11:29:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:50' 2024-09-16T11:29:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:29:50' 2024-09-16T11:29:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:29:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:50' 2024-09-16T11:29:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":51},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":54}}44464452mirror.0smithi2042024-09-16 11:29:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:29:50' 2024-09-16T11:29:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:50' 2024-09-16T11:29:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:29:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:29:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:29:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:29:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:29:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:29:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:29:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:29:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:29:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:29:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:29:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:29:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:29:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:29:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:29:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:29:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:29:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:29:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:29:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:29:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:29:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:29:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:29:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:29:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:29:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:55 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:29:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:29:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:29:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:56' 2024-09-16T11:29:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:29:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:29:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:29:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:29:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:29:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:29:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:29:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:29:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:29:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:29:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:55' 2024-09-16T11:29:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:29:55' 2024-09-16T11:29:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:29:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:29:55' 2024-09-16T11:29:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:29:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:29:59.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:29:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:29:55cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:29:56' 2024-09-16T11:29:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:29:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:29:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:29:55' 2024-09-16T11:29:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:29:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:29:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:29:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:29:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:29:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:29:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:29:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:29:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:29:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:29:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:29:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:29:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:29:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:29:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:30:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:56 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:29:55' 2024-09-16T11:30:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:30:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:00 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:00' 2024-09-16T11:30:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:00' 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:30:00' 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:00' 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:30:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:30:00' 2024-09-16T11:30:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:00' 2024-09-16T11:30:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:30:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:30:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:30:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:02.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:03.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:30:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:30:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:30:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:30:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:30:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:30:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:30:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:30:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:30:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:30:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:30:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:30:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:30:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:30:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:30:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:30:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:30:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:30:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:30:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:30:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:30:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:30:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:30:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:30:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:30:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:30:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:30:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:30:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:30:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:30:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=58, entry_tid=0' 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:30:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=58, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=58, entry_tid=0' 2024-09-16T11:30:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=58, entry_tid=0' -a 'object_number=0, tag_tid=58, entry_tid=0' = 'object_number=0, tag_tid=58, entry_tid=0' 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:05 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2024-09-16T11:30:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:05' 2024-09-16T11:30:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:30:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:30:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:30:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:07.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:05' 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:30:05' 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:05' 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:30:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}2024-09-16 11:30:05' 2024-09-16T11:30:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:05' 2024-09-16T11:30:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:30:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:07.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:30:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:30:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:05 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:05' 2024-09-16T11:30:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:30:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:30:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}} 2024-09-16T11:30:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:05' 2024-09-16T11:30:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:30:05' 2024-09-16T11:30:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:05' 2024-09-16T11:30:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":55},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":58}}42884295mirror.0smithi2042024-09-16 11:30:05cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:30:05' 2024-09-16T11:30:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:05' 2024-09-16T11:30:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:30:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:30:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:30:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:30:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:30:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:30:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:30:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:30:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:10 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:10' 2024-09-16T11:30:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:10' 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:30:10' 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:10' 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:30:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:30:10' 2024-09-16T11:30:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:10' 2024-09-16T11:30:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:10 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:10' 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:10' 2024-09-16T11:30:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:30:10' 2024-09-16T11:30:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:10' 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:30:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:30:10' 2024-09-16T11:30:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:10' 2024-09-16T11:30:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:30:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:30:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:30:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:30:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:30:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:30:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:30:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:30:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:30:17.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:30:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:30:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:30:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:30:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:30:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:30:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:30:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:30:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:30:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=59, entry_tid=0' 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:30:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=59, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=59, entry_tid=0' 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=59, entry_tid=0' -a 'object_number=0, tag_tid=59, entry_tid=0' = 'object_number=0, tag_tid=59, entry_tid=0' 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:30:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:15 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:15' 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:30:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:30:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:30:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:15' 2024-09-16T11:30:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:30:15' 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:15' 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:30:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}2024-09-16 11:30:15' 2024-09-16T11:30:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:15' 2024-09-16T11:30:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:30:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:30:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:30:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:18.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:30:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:15 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:15' 2024-09-16T11:30:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:30:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:30:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}} 2024-09-16T11:30:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:15' 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:30:15' 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:15' 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":56},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":59}}44464452mirror.0smithi2042024-09-16 11:30:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:30:15' 2024-09-16T11:30:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:15' 2024-09-16T11:30:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:30:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:30:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:30:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:30:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:30:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:30:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:30:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:21.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:21.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:21.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:21.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:30:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:20 2024-09-16T11:30:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:30:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:21' 2024-09-16T11:30:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:20' 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:30:20' 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:20' 2024-09-16T11:30:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:30:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:30:21' 2024-09-16T11:30:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:20' 2024-09-16T11:30:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:21 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:25' 2024-09-16T11:30:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:30:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:25 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:25' 2024-09-16T11:30:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:25' 2024-09-16T11:30:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:30:25' 2024-09-16T11:30:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:25' 2024-09-16T11:30:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:30:25cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:30:25' 2024-09-16T11:30:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:25' 2024-09-16T11:30:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:30:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:30:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:30:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:30:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:30:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:30:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:30:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:30:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:30:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:30:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:30:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=63, entry_tid=0' 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:30:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:30:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=63, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=63, entry_tid=0' 2024-09-16T11:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=63, entry_tid=0' -a 'object_number=0, tag_tid=63, entry_tid=0' = 'object_number=0, tag_tid=63, entry_tid=0' 2024-09-16T11:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:30 2024-09-16T11:30:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:30' 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:30:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:30:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:30' 2024-09-16T11:30:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:30:30' 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:30' 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:30:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}2024-09-16 11:30:30' 2024-09-16T11:30:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:30' 2024-09-16T11:30:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:32.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:30:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:32.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:30:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:30 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:30' 2024-09-16T11:30:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:30:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:30:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}} 2024-09-16T11:30:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:30' 2024-09-16T11:30:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:30:30' 2024-09-16T11:30:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:30' 2024-09-16T11:30:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:33.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":60},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":63}}42884295mirror.0smithi2042024-09-16 11:30:30cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:30:30' 2024-09-16T11:30:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:30' 2024-09-16T11:30:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:30:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:30:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:30:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:30:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:30:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:30:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:30:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:30:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:30:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:35 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:35' 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:35' 2024-09-16T11:30:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:30:35' 2024-09-16T11:30:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:35' 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:30:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:30:35' 2024-09-16T11:30:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:35' 2024-09-16T11:30:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:35 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:35' 2024-09-16T11:30:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:35' 2024-09-16T11:30:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:30:35' 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:35' 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:39.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:30:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:30:35' 2024-09-16T11:30:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:35' 2024-09-16T11:30:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:30:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:30:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:30:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:30:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:30:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:30:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:30:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:30:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:30:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:30:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:30:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:30:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:30:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:30:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=64, entry_tid=0' 2024-09-16T11:30:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:30:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:30:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=64, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:30:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=64, entry_tid=0' 2024-09-16T11:30:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=64, entry_tid=0' -a 'object_number=0, tag_tid=64, entry_tid=0' = 'object_number=0, tag_tid=64, entry_tid=0' 2024-09-16T11:30:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:30:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:40 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:40' 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:30:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:30:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:30:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:40' 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:30:40' 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:40' 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:30:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}2024-09-16 11:30:40' 2024-09-16T11:30:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:40' 2024-09-16T11:30:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:30:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:30:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:30:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:40 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:40' 2024-09-16T11:30:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:30:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:30:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}} 2024-09-16T11:30:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:40' 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:30:40' 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:40' 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":61},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":64}}44464452mirror.0smithi2042024-09-16 11:30:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:30:40' 2024-09-16T11:30:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:40' 2024-09-16T11:30:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:30:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:30:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:30:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:30:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:30:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:30:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:30:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:30:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:30:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:30:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:45 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:30:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:46' 2024-09-16T11:30:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:45' 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:30:45' 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:45' 2024-09-16T11:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:30:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:30:46' 2024-09-16T11:30:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:45' 2024-09-16T11:30:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:49.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:46 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:45' 2024-09-16T11:30:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:30:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:50 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:50' 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:30:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:30:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:50' 2024-09-16T11:30:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:30:50' 2024-09-16T11:30:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:50' 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:30:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:30:50' 2024-09-16T11:30:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:50' 2024-09-16T11:30:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:30:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:30:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:30:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:30:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:30:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:30:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:30:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:30:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:30:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:30:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:30:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:30:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:30:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:30:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:30:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:30:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:30:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:30:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:30:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:30:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:30:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:30:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:30:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:30:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:30:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=68, entry_tid=0' 2024-09-16T11:30:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:30:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:30:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:30:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:30:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:30:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:30:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:30:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:30:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:30:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:30:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:30:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:30:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:30:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=68, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:30:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:30:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=68, entry_tid=0' 2024-09-16T11:30:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=68, entry_tid=0' -a 'object_number=0, tag_tid=68, entry_tid=0' = 'object_number=0, tag_tid=68, entry_tid=0' 2024-09-16T11:30:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:30:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:56.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:30:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:30:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:55 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:55' 2024-09-16T11:30:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:30:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:30:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:30:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:55' 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:30:55' 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:55' 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:30:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}2024-09-16 11:30:55' 2024-09-16T11:30:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:55' 2024-09-16T11:30:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:30:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:30:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:30:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:55 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:30:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:30:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:30:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:30:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:30:55' 2024-09-16T11:30:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:30:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:30:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:30:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}} 2024-09-16T11:30:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:30:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:30:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:30:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:30:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:30:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:55' 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:30:55' 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:30:55' 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:30:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:30:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":65},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":68}}42884295mirror.0smithi2042024-09-16 11:30:55cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:30:55' 2024-09-16T11:30:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:30:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:30:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:30:55' 2024-09-16T11:30:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:30:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:30:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:30:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:30:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:30:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:30:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:30:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:30:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:30:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:30:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:30:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:30:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:30:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:30:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:30:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:30:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:30:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:30:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:30:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:30:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:30:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:30:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:30:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:30:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:30:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:30:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:30:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:30:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:30:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:30:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:00.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:31:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:00 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:00' 2024-09-16T11:31:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:03.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:00' 2024-09-16T11:31:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:31:00' 2024-09-16T11:31:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:00' 2024-09-16T11:31:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:31:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:31:00' 2024-09-16T11:31:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:00' 2024-09-16T11:31:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:00 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:00' 2024-09-16T11:31:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:00' 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:31:00' 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:00' 2024-09-16T11:31:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:31:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:31:00' 2024-09-16T11:31:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:00' 2024-09-16T11:31:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:31:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:31:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:31:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:31:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:31:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:05.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:31:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:31:07.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:31:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:31:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:31:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:31:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:31:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:31:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:31:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:31:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=69, entry_tid=0' 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:08.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:31:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=69, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=69, entry_tid=0' 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=69, entry_tid=0' -a 'object_number=0, tag_tid=69, entry_tid=0' = 'object_number=0, tag_tid=69, entry_tid=0' 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:05 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2024-09-16T11:31:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:05' 2024-09-16T11:31:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:31:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:31:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:31:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:05' 2024-09-16T11:31:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:31:05' 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:05' 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:09.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:31:05cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}2024-09-16 11:31:05' 2024-09-16T11:31:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:05' 2024-09-16T11:31:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:31:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:09.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:31:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2024-09-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:05 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:05' 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}} 2024-09-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:05' 2024-09-16T11:31:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:31:05' 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:05' 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":66},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":69}}44464452mirror.0smithi2042024-09-16 11:31:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:31:05' 2024-09-16T11:31:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:05' 2024-09-16T11:31:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:31:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:31:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:31:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:31:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:31:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:10 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:11' 2024-09-16T11:31:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:10' 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:31:10' 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:10' 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:31:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:31:11' 2024-09-16T11:31:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:10' 2024-09-16T11:31:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:11 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:10' 2024-09-16T11:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:31:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:11 2024-09-16T11:31:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:15' 2024-09-16T11:31:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:31:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:15 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:15' 2024-09-16T11:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:15' 2024-09-16T11:31:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:31:15' 2024-09-16T11:31:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:15' 2024-09-16T11:31:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:31:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:31:15' 2024-09-16T11:31:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:15' 2024-09-16T11:31:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:31:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:31:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:31:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:31:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:31:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:31:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:31:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:31:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:31:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:31:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:31:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:31:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:31:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:31:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=73, entry_tid=0' 2024-09-16T11:31:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:31:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=73, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=73, entry_tid=0' 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=73, entry_tid=0' -a 'object_number=0, tag_tid=73, entry_tid=0' = 'object_number=0, tag_tid=73, entry_tid=0' 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:31:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:20 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:20' 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:31:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:31:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:20' 2024-09-16T11:31:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:31:20' 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:20' 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:31:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}2024-09-16 11:31:20' 2024-09-16T11:31:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:20' 2024-09-16T11:31:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:31:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:31:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:20 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:20' 2024-09-16T11:31:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:31:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:31:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}} 2024-09-16T11:31:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:23.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:23.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:20' 2024-09-16T11:31:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:31:20' 2024-09-16T11:31:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:20' 2024-09-16T11:31:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":70},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":73}}42884295mirror.0smithi2042024-09-16 11:31:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:31:20' 2024-09-16T11:31:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:20' 2024-09-16T11:31:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:31:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:31:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:31:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:31:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:31:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:25 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:25' 2024-09-16T11:31:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:25' 2024-09-16T11:31:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:31:25' 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:25' 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:31:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:31:25' 2024-09-16T11:31:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:25' 2024-09-16T11:31:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:28.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:25 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:25' 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:29.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:25' 2024-09-16T11:31:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:31:25' 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:25' 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:31:25cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:31:25' 2024-09-16T11:31:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:25' 2024-09-16T11:31:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:31:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:31:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:31:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:31:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:31:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:30.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:31:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:31:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:31:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:31:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:31:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:31:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:31:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:31:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:31:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:31:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=74, entry_tid=0' 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:31:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:31:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=74, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=74, entry_tid=0' 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=74, entry_tid=0' -a 'object_number=0, tag_tid=74, entry_tid=0' = 'object_number=0, tag_tid=74, entry_tid=0' 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:30 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:30' 2024-09-16T11:31:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:31:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:31:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:31:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:30' 2024-09-16T11:31:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:31:30' 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:30' 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:31:30cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}2024-09-16 11:31:30' 2024-09-16T11:31:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:30' 2024-09-16T11:31:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:31:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:31:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:31:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:31:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:30 2024-09-16T11:31:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:30' 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}} 2024-09-16T11:31:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:30' 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:31:30' 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:30' 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":71},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":74}}44464452mirror.0smithi2042024-09-16 11:31:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:31:30' 2024-09-16T11:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:30' 2024-09-16T11:31:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:31:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:31:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:31:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:31:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:31:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:31:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:31:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:31:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:39.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:35 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:36' 2024-09-16T11:31:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:35' 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:31:35' 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:35' 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:31:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:31:36' 2024-09-16T11:31:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:35' 2024-09-16T11:31:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:39.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:36 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:35' 2024-09-16T11:31:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:31:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:40 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:40' 2024-09-16T11:31:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:40' 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:31:40' 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:40' 2024-09-16T11:31:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:31:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:31:40' 2024-09-16T11:31:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:40' 2024-09-16T11:31:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:31:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:31:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:31:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:31:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:31:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:31:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:45.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:31:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:31:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:31:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:31:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:31:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:31:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:31:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:31:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:31:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:31:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:31:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:31:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:31:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:31:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=75, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0' 2024-09-16T11:31:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:31:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:31:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:31:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:31:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=75, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=75, entry_tid=0' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=75, entry_tid=0' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=75, entry_tid=0' '!=' '' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=75, entry_tid=0' = '' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=75, entry_tid=0' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=78, entry_tid=0' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:31:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:31:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=78 2024-09-16T11:31:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=75, entry_tid=0' 2024-09-16T11:31:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:31:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:31:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=75 2024-09-16T11:31:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=78, entry_tid=0' 2024-09-16T11:31:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:31:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:31:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-09-16T11:31:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=75, entry_tid=0' 2024-09-16T11:31:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:31:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:31:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:31:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 78 = 75 -a 0 -le 0 2024-09-16T11:31:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:31:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:31:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:31:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:31:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:31:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:31:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:31:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:31:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:31:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=78, entry_tid=0' 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:31:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:31:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=78, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:31:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=78, entry_tid=0' 2024-09-16T11:31:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=78, entry_tid=0' -a 'object_number=0, tag_tid=78, entry_tid=0' = 'object_number=0, tag_tid=78, entry_tid=0' 2024-09-16T11:31:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:45 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2024-09-16T11:31:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:45' 2024-09-16T11:31:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:31:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:31:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:31:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:45' 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:31:45' 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:45' 2024-09-16T11:31:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:31:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}2024-09-16 11:31:45' 2024-09-16T11:31:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:45' 2024-09-16T11:31:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:31:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:31:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2024-09-16T11:31:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:45 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:45' 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:31:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}} 2024-09-16T11:31:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:45' 2024-09-16T11:31:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:31:45' 2024-09-16T11:31:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:45' 2024-09-16T11:31:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":75},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":78}}42884295mirror.0smithi2042024-09-16 11:31:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:31:45' 2024-09-16T11:31:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:45' 2024-09-16T11:31:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:31:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:31:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:31:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:31:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:31:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:31:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:31:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:31:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:31:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:31:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:31:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:31:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:31:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:31:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:31:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:50 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:50' 2024-09-16T11:31:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:53.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:50' 2024-09-16T11:31:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:31:50' 2024-09-16T11:31:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:50' 2024-09-16T11:31:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:31:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:31:50' 2024-09-16T11:31:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:50' 2024-09-16T11:31:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:31:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:50 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:50' 2024-09-16T11:31:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:50' 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:31:50' 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:50' 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:31:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:31:50' 2024-09-16T11:31:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:50' 2024-09-16T11:31:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:31:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:31:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:31:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:31:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:31:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:31:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:31:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:31:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:31:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:31:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:31:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:31:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:31:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:31:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:31:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:31:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:31:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:57.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:31:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:31:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:31:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:31:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:31:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:31:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:31:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:31:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:31:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:31:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:31:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:31:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:31:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:31:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:31:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:31:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:31:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:31:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:31:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:31:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:31:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:31:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:31:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:31:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:31:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:31:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:31:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:31:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:31:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:31:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:31:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:31:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:31:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:31:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:31:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:31:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=79, entry_tid=0' 2024-09-16T11:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:31:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:31:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:31:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:31:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:31:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:31:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=79, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:31:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=79, entry_tid=0' 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=79, entry_tid=0' -a 'object_number=0, tag_tid=79, entry_tid=0' = 'object_number=0, tag_tid=79, entry_tid=0' 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:55 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:31:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:31:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2024-09-16T11:31:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:55' 2024-09-16T11:31:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:31:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:31:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:31:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:31:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:31:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:31:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:31:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:31:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:31:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:55' 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:31:55' 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:55' 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:31:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:31:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:31:55cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}2024-09-16 11:31:55' 2024-09-16T11:31:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:31:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:31:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:55' 2024-09-16T11:31:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:31:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:31:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:31:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:31:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:31:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:31:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:31:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:31:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:31:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:32:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:55 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:31:55' 2024-09-16T11:32:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:32:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:32:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}} 2024-09-16T11:32:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:55' 2024-09-16T11:32:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:31:55' 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:31:55' 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":76},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":79}}44464452mirror.0smithi2042024-09-16 11:31:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:31:55' 2024-09-16T11:32:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:31:55' 2024-09-16T11:32:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:32:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:32:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:32:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:32:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:32:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:32:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:01.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:32:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:00 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:02' 2024-09-16T11:32:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:00' 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:32:00' 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:00' 2024-09-16T11:32:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:32:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:32:02' 2024-09-16T11:32:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:00' 2024-09-16T11:32:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:02 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:05' 2024-09-16T11:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:32:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:05 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:05' 2024-09-16T11:32:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:05' 2024-09-16T11:32:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:32:05' 2024-09-16T11:32:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:05' 2024-09-16T11:32:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:32:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:32:05' 2024-09-16T11:32:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:05' 2024-09-16T11:32:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:32:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:32:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:32:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:32:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:09.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:32:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:32:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:32:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:32:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:32:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:32:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:32:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:32:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:32:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:32:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:32:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=83, entry_tid=0' 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:32:11.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:11.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=83, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=83, entry_tid=0' 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=83, entry_tid=0' -a 'object_number=0, tag_tid=83, entry_tid=0' = 'object_number=0, tag_tid=83, entry_tid=0' 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:10 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:10' 2024-09-16T11:32:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:32:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:32:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:32:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:10' 2024-09-16T11:32:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:32:10' 2024-09-16T11:32:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:10' 2024-09-16T11:32:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:32:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}2024-09-16 11:32:10' 2024-09-16T11:32:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:10' 2024-09-16T11:32:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:32:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:32:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:32:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:32:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:10 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:10' 2024-09-16T11:32:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:32:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:32:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}} 2024-09-16T11:32:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:10' 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:32:10' 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:10' 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":80},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":83}}42884295mirror.0smithi2042024-09-16 11:32:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:32:10' 2024-09-16T11:32:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:10' 2024-09-16T11:32:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:32:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:32:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:32:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:32:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:32:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:32:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:32:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:32:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:32:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:15 2024-09-16T11:32:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:15' 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:15' 2024-09-16T11:32:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:32:15' 2024-09-16T11:32:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:15' 2024-09-16T11:32:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:32:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:32:15' 2024-09-16T11:32:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:15' 2024-09-16T11:32:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:15 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:15' 2024-09-16T11:32:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:15' 2024-09-16T11:32:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:32:15' 2024-09-16T11:32:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:15' 2024-09-16T11:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:32:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:32:15' 2024-09-16T11:32:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:15' 2024-09-16T11:32:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:32:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:32:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:32:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:32:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:22.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:32:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:32:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:32:22.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:32:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:32:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:32:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:32:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:32:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:32:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:32:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:32:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=84, entry_tid=0' 2024-09-16T11:32:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:32:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:32:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=84, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=84, entry_tid=0' 2024-09-16T11:32:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=84, entry_tid=0' -a 'object_number=0, tag_tid=84, entry_tid=0' = 'object_number=0, tag_tid=84, entry_tid=0' 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:20 2024-09-16T11:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2024-09-16T11:32:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:20' 2024-09-16T11:32:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:32:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:32:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:32:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:20' 2024-09-16T11:32:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:32:20' 2024-09-16T11:32:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:20' 2024-09-16T11:32:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:32:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}2024-09-16 11:32:20' 2024-09-16T11:32:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:20' 2024-09-16T11:32:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:32:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:32:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2024-09-16T11:32:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:20 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:20' 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:32:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:32:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}} 2024-09-16T11:32:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:20' 2024-09-16T11:32:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:32:20' 2024-09-16T11:32:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:20' 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":81},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":84}}44464452mirror.0smithi2042024-09-16 11:32:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:32:20' 2024-09-16T11:32:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:20' 2024-09-16T11:32:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:32:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:32:25.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:25.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:32:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:32:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:25 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:26' 2024-09-16T11:32:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:25' 2024-09-16T11:32:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:32:25' 2024-09-16T11:32:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:25' 2024-09-16T11:32:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:32:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:32:26' 2024-09-16T11:32:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:25' 2024-09-16T11:32:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:27.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:26 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:25' 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:32:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:26 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:30' 2024-09-16T11:32:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:32:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:32:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:30 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:30' 2024-09-16T11:32:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:30' 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:32:30' 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:30' 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:34.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:32:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:32:30' 2024-09-16T11:32:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:30' 2024-09-16T11:32:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:32:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:32:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:32:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:32:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:32:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:32:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:32:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:32:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:32:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:32:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:32:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:32:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:32:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=88, entry_tid=0' 2024-09-16T11:32:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:32:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=88, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:32:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=88, entry_tid=0' 2024-09-16T11:32:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=88, entry_tid=0' -a 'object_number=0, tag_tid=88, entry_tid=0' = 'object_number=0, tag_tid=88, entry_tid=0' 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:35 2024-09-16T11:32:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2024-09-16T11:32:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:35' 2024-09-16T11:32:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:32:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:32:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:32:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:35' 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:32:35' 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:35' 2024-09-16T11:32:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:32:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}2024-09-16 11:32:35' 2024-09-16T11:32:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:35' 2024-09-16T11:32:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:32:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:32:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:35 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:35' 2024-09-16T11:32:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:32:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:32:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}} 2024-09-16T11:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:35' 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:32:35' 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:35' 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":85},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":88}}42884295mirror.0smithi2042024-09-16 11:32:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:32:35' 2024-09-16T11:32:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:35' 2024-09-16T11:32:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:32:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:32:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:32:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:38.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:32:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:32:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:32:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:40 2024-09-16T11:32:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:40' 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:40' 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:32:40' 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:40' 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:32:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:32:40' 2024-09-16T11:32:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:40' 2024-09-16T11:32:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:40 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:40' 2024-09-16T11:32:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:40' 2024-09-16T11:32:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:32:40' 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:40' 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:32:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:32:40' 2024-09-16T11:32:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:40' 2024-09-16T11:32:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:32:44.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:32:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:32:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:32:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:32:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:45.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:32:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:47.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:32:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:32:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:32:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:32:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:32:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:32:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:32:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:32:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=89, entry_tid=0' 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:32:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:32:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:32:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:32:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=89, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:32:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=89, entry_tid=0' 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=89, entry_tid=0' -a 'object_number=0, tag_tid=89, entry_tid=0' = 'object_number=0, tag_tid=89, entry_tid=0' 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:45 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:45' 2024-09-16T11:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:32:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:32:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:32:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:45' 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:32:45' 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:45' 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:32:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}2024-09-16 11:32:45' 2024-09-16T11:32:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:45' 2024-09-16T11:32:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:32:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:32:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:32:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:32:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:32:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:45 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:45' 2024-09-16T11:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}} 2024-09-16T11:32:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:45' 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:32:45' 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:32:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:45' 2024-09-16T11:32:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":86},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":89}}44464452mirror.0smithi2042024-09-16 11:32:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:32:45' 2024-09-16T11:32:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:45' 2024-09-16T11:32:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:32:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:32:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:32:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:32:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:32:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:32:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:32:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:32:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:32:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:32:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:32:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:32:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:32:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:32:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:32:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:32:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:32:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:32:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:50 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:51' 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:50' 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:32:50' 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:50' 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:32:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:32:51' 2024-09-16T11:32:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:50' 2024-09-16T11:32:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:51 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:55' 2024-09-16T11:32:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:32:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:32:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:32:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:55 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:32:55' 2024-09-16T11:32:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:32:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:32:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:32:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:32:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:32:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:32:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:32:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:32:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:32:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:32:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:55' 2024-09-16T11:32:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:32:55' 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:32:55' 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:32:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:32:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:32:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:32:55' 2024-09-16T11:32:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:32:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:32:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:32:55' 2024-09-16T11:32:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:32:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:32:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:32:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:32:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:32:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:32:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:32:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:32:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:32:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:32:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:32:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:32:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:32:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:32:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:32:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:32:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:32:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:32:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:32:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:32:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:32:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:32:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:32:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:32:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:32:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:32:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:32:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:32:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:32:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:32:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:32:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:32:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:32:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:32:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:32:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:33:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:01.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:33:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:33:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:33:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:33:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:33:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:33:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:33:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:33:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:33:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:33:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:33:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:33:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=93, entry_tid=0' 2024-09-16T11:33:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:33:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:33:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=93, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=93, entry_tid=0' 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=93, entry_tid=0' -a 'object_number=0, tag_tid=93, entry_tid=0' = 'object_number=0, tag_tid=93, entry_tid=0' 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:33:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:00 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:00' 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:33:02.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:33:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:02.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:00' 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:33:00' 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:00' 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:33:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}2024-09-16 11:33:00' 2024-09-16T11:33:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:00' 2024-09-16T11:33:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:33:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:33:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:00 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:00' 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:33:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:33:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}} 2024-09-16T11:33:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:00' 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:33:00' 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:00' 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":90},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":93}}42884295mirror.0smithi2042024-09-16 11:33:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:33:00' 2024-09-16T11:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:00' 2024-09-16T11:33:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:33:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:33:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:33:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:33:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:05 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:05' 2024-09-16T11:33:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:05' 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:33:05' 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:05' 2024-09-16T11:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:33:05cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:33:05' 2024-09-16T11:33:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:05' 2024-09-16T11:33:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:33:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:05 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:09.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:05' 2024-09-16T11:33:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:05' 2024-09-16T11:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:33:05' 2024-09-16T11:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:05' 2024-09-16T11:33:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:33:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:33:05' 2024-09-16T11:33:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:05' 2024-09-16T11:33:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:33:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:33:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:33:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:33:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:33:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:33:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:33:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:12.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:33:12.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:33:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:33:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:33:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:33:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:33:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:33:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:33:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=94, entry_tid=0' 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:33:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=94, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=94, entry_tid=0' 2024-09-16T11:33:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=94, entry_tid=0' -a 'object_number=0, tag_tid=94, entry_tid=0' = 'object_number=0, tag_tid=94, entry_tid=0' 2024-09-16T11:33:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:33:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:10 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:10' 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:33:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:33:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:10' 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:33:10' 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:10' 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:33:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}2024-09-16 11:33:10' 2024-09-16T11:33:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:10' 2024-09-16T11:33:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:33:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:33:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:10 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:10' 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:33:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:33:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}} 2024-09-16T11:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:15.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:10' 2024-09-16T11:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:33:10' 2024-09-16T11:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:10' 2024-09-16T11:33:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":91},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":94}}44464452mirror.0smithi2042024-09-16 11:33:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:33:10' 2024-09-16T11:33:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:10' 2024-09-16T11:33:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:33:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:33:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:33:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:33:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:33:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:33:16.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:33:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:33:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:33:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:33:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:15 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:16' 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:15' 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:33:15' 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:15' 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:33:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:33:16' 2024-09-16T11:33:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:15' 2024-09-16T11:33:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:16 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:15' 2024-09-16T11:33:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:33:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:16 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:20' 2024-09-16T11:33:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:33:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:33:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:20 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:20' 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:20' 2024-09-16T11:33:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:33:20' 2024-09-16T11:33:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:20' 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:33:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:33:20' 2024-09-16T11:33:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:20' 2024-09-16T11:33:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:33:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:33:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:33:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:33:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:33:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:33:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:33:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:33:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:33:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:33:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:33:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:33:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:33:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:33:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:33:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:33:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=98, entry_tid=0' 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:33:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=98, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:26.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=98, entry_tid=0' 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=98, entry_tid=0' -a 'object_number=0, tag_tid=98, entry_tid=0' = 'object_number=0, tag_tid=98, entry_tid=0' 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:26.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:26.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:25 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2024-09-16T11:33:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:25' 2024-09-16T11:33:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:33:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:33:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:33:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:25' 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:33:25' 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:25' 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:27.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:33:25cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}2024-09-16 11:33:25' 2024-09-16T11:33:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:25' 2024-09-16T11:33:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:33:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:33:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:33:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:27.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:33:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:25 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:25' 2024-09-16T11:33:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:33:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:33:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}} 2024-09-16T11:33:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:25' 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:33:25' 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:25' 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":95},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":98}}42884295mirror.0smithi2042024-09-16 11:33:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:33:25' 2024-09-16T11:33:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:25' 2024-09-16T11:33:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:33:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:33:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:33:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:33:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:32.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:30 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:30' 2024-09-16T11:33:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:30' 2024-09-16T11:33:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:33:30' 2024-09-16T11:33:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:30' 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:33:30cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:33:30' 2024-09-16T11:33:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:30' 2024-09-16T11:33:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:30 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:30' 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:34.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:30' 2024-09-16T11:33:34.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:33:30' 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:30' 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:34.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:33:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:33:30' 2024-09-16T11:33:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:30' 2024-09-16T11:33:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:33:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:33:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:33:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:33:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:33:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:35.581 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:35.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:33:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:33:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:33:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:33:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:33:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:33:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:33:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:33:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:33:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:33:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:33:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:33:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:33:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=99, entry_tid=0' 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:33:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:33:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=99, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:33:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=99, entry_tid=0' 2024-09-16T11:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=99, entry_tid=0' -a 'object_number=0, tag_tid=99, entry_tid=0' = 'object_number=0, tag_tid=99, entry_tid=0' 2024-09-16T11:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:35 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:35' 2024-09-16T11:33:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:33:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:33:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:33:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:35' 2024-09-16T11:33:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:33:35' 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:35' 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:33:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}2024-09-16 11:33:35' 2024-09-16T11:33:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:35' 2024-09-16T11:33:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:33:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:33:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:39.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:33:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:35 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:35' 2024-09-16T11:33:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:33:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:33:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}} 2024-09-16T11:33:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:35' 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:33:35' 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:35' 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":96},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":99}}44464452mirror.0smithi2042024-09-16 11:33:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:33:35' 2024-09-16T11:33:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:35' 2024-09-16T11:33:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:33:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:33:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:33:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:33:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:33:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:33:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:33:40.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:33:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:40.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:33:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:33:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:33:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:33:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:33:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:33:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:40 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:41' 2024-09-16T11:33:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:40' 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:33:40' 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:40' 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:33:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:33:41' 2024-09-16T11:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:40' 2024-09-16T11:33:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:41 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:40' 2024-09-16T11:33:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:33:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:33:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:41 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:45' 2024-09-16T11:33:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:33:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:33:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:45 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:45' 2024-09-16T11:33:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:45' 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:33:45' 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:45' 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:33:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:33:45' 2024-09-16T11:33:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:45' 2024-09-16T11:33:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:33:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:33:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:33:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:33:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:33:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:33:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:33:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:33:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:33:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:33:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:33:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:33:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:33:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:33:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:33:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:33:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:33:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=103, entry_tid=0' 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:33:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:33:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:33:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:33:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:33:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:33:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:33:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:33:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:33:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:33:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:33:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=103, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:33:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:33:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=103, entry_tid=0' 2024-09-16T11:33:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=103, entry_tid=0' -a 'object_number=0, tag_tid=103, entry_tid=0' = 'object_number=0, tag_tid=103, entry_tid=0' 2024-09-16T11:33:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:33:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:50 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:50' 2024-09-16T11:33:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:33:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:33:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:33:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:50' 2024-09-16T11:33:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:33:50' 2024-09-16T11:33:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:33:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:50' 2024-09-16T11:33:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:33:50cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}2024-09-16 11:33:50' 2024-09-16T11:33:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:50' 2024-09-16T11:33:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:33:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:33:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:33:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:50 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:50' 2024-09-16T11:33:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:33:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:33:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}} 2024-09-16T11:33:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:50' 2024-09-16T11:33:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:33:50' 2024-09-16T11:33:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:53.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:33:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:50' 2024-09-16T11:33:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":100},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":103}}42884295mirror.0smithi2042024-09-16 11:33:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:33:50' 2024-09-16T11:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:50' 2024-09-16T11:33:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:33:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:33:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:33:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:33:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:33:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:33:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:33:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:33:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:33:54.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:33:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:33:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:33:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:33:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:33:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:33:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:33:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:33:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:33:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:33:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:33:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:33:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:33:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:33:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:33:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:33:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:33:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:33:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:33:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:33:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:33:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:33:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:55 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:33:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:55' 2024-09-16T11:33:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:55' 2024-09-16T11:33:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:33:55' 2024-09-16T11:33:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:55' 2024-09-16T11:33:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:33:55cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:33:55' 2024-09-16T11:33:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:55' 2024-09-16T11:33:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:33:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:55 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:33:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:33:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:33:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:33:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:33:55' 2024-09-16T11:33:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:33:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:33:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:33:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:33:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:33:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:33:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:33:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:33:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:33:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:33:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:55' 2024-09-16T11:33:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:33:55' 2024-09-16T11:33:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:33:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:33:55' 2024-09-16T11:33:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:33:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:33:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:33:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:33:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:33:55' 2024-09-16T11:33:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:33:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:33:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:33:55' 2024-09-16T11:33:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:33:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:33:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:33:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:33:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:33:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:33:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:33:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:33:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:33:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:33:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:33:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:34:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:34:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:02.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:02.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:34:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:02.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:34:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:34:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:02.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:02.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:34:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=104, entry_tid=0' 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=104, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=104, entry_tid=0' 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=104, entry_tid=0' -a 'object_number=0, tag_tid=104, entry_tid=0' = 'object_number=0, tag_tid=104, entry_tid=0' 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:34:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:00 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:00' 2024-09-16T11:34:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:34:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:34:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:34:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:00' 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:34:00' 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:00' 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:34:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}2024-09-16 11:34:00' 2024-09-16T11:34:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:00' 2024-09-16T11:34:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:34:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:34:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:00 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:00' 2024-09-16T11:34:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:34:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:34:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}} 2024-09-16T11:34:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:05.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:00' 2024-09-16T11:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:34:00' 2024-09-16T11:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:00' 2024-09-16T11:34:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":101},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":104}}44464452mirror.0smithi2042024-09-16 11:34:00cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:34:00' 2024-09-16T11:34:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:00' 2024-09-16T11:34:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:34:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:34:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:34:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:34:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:34:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:34:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:08.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:05 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:06' 2024-09-16T11:34:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:05' 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:34:05' 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:05' 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:34:05cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:34:06' 2024-09-16T11:34:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:05' 2024-09-16T11:34:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:06 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:10' 2024-09-16T11:34:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:34:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:10 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:10' 2024-09-16T11:34:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:10' 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:34:10' 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:10' 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:34:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:34:10' 2024-09-16T11:34:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:10' 2024-09-16T11:34:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:34:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:34:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:34:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:34:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:34:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:34:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:14.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:34:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:34:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:34:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=105, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=108, entry_tid=0' 2024-09-16T11:34:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=105, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=105, entry_tid=0' 2024-09-16T11:34:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=108, entry_tid=0' -a 'object_number=0, tag_tid=108, entry_tid=0' = 'object_number=0, tag_tid=105, entry_tid=0' 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=105, entry_tid=0' '!=' '' 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=105, entry_tid=0' = '' 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=105, entry_tid=0' 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=108, entry_tid=0' 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=108 2024-09-16T11:34:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=105, entry_tid=0' 2024-09-16T11:34:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=105 2024-09-16T11:34:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=108, entry_tid=0' 2024-09-16T11:34:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-09-16T11:34:16.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=105, entry_tid=0' 2024-09-16T11:34:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:34:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 108 = 105 -a 0 -le 0 2024-09-16T11:34:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:34:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:34:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:34:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:34:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=108, entry_tid=0' 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=108, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=108, entry_tid=0' 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=108, entry_tid=0' -a 'object_number=0, tag_tid=108, entry_tid=0' = 'object_number=0, tag_tid=108, entry_tid=0' 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:34:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:15 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2024-09-16T11:34:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:15' 2024-09-16T11:34:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:34:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:34:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:34:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:15' 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:34:15' 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:15' 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:34:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}2024-09-16 11:34:15' 2024-09-16T11:34:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:15' 2024-09-16T11:34:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:34:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:34:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:34:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:15 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:15' 2024-09-16T11:34:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:34:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:34:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}} 2024-09-16T11:34:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:15' 2024-09-16T11:34:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:34:15' 2024-09-16T11:34:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:15' 2024-09-16T11:34:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":105},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":108}}42884295mirror.0smithi2042024-09-16 11:34:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:34:15' 2024-09-16T11:34:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:15' 2024-09-16T11:34:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:34:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:34:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:34:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:34:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:34:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:34:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:34:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:34:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:34:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:34:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:20 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:20' 2024-09-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:20' 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:34:20' 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:20' 2024-09-16T11:34:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:34:20cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:34:20' 2024-09-16T11:34:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:20' 2024-09-16T11:34:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:23.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:20 2024-09-16T11:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:20' 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:20' 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:34:20' 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:20' 2024-09-16T11:34:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:34:20cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:34:20' 2024-09-16T11:34:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:20' 2024-09-16T11:34:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:34:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:34:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:34:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:34:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:34:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:34:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:34:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:34:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:34:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:34:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:34:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:34:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:34:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=106, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=109, entry_tid=0' 2024-09-16T11:34:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:34:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=106, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=106, entry_tid=0' 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=109, entry_tid=0' -a 'object_number=0, tag_tid=109, entry_tid=0' = 'object_number=0, tag_tid=106, entry_tid=0' 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=106, entry_tid=0' '!=' '' 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=106, entry_tid=0' = '' 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=106, entry_tid=0' 2024-09-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=109, entry_tid=0' 2024-09-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=109 2024-09-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=106, entry_tid=0' 2024-09-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=106 2024-09-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=109, entry_tid=0' 2024-09-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-09-16T11:34:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=106, entry_tid=0' 2024-09-16T11:34:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:34:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 109 = 106 -a 0 -le 0 2024-09-16T11:34:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:34:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=109, entry_tid=0' 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=109, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=109, entry_tid=0' 2024-09-16T11:34:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=109, entry_tid=0' -a 'object_number=0, tag_tid=109, entry_tid=0' = 'object_number=0, tag_tid=109, entry_tid=0' 2024-09-16T11:34:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:34:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:25 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:25' 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:34:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:34:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:34:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:25' 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:34:25' 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:25' 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:27.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:34:25cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}2024-09-16 11:34:25' 2024-09-16T11:34:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:25' 2024-09-16T11:34:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:34:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:34:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:25 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:25' 2024-09-16T11:34:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:34:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:34:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}} 2024-09-16T11:34:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:25' 2024-09-16T11:34:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:34:25' 2024-09-16T11:34:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:25' 2024-09-16T11:34:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":106},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":109}}44464452mirror.0smithi2042024-09-16 11:34:25cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:34:25' 2024-09-16T11:34:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:25' 2024-09-16T11:34:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:34:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:34:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:34:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:34:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:34:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:34:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:34:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:32.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:34:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:34:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:30 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:30' 2024-09-16T11:34:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:30' 2024-09-16T11:34:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:34:30' 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:30' 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:34:30cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:34:30' 2024-09-16T11:34:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:30' 2024-09-16T11:34:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:34:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:30 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:30' 2024-09-16T11:34:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:30' 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:34:30' 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:30' 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:34:30cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:34:30' 2024-09-16T11:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:30' 2024-09-16T11:34:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:34:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:34:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:34:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:34:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:34:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:34:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:34:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:34:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=110, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=113, entry_tid=0' 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:36.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=110, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=110, entry_tid=0' 2024-09-16T11:34:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=113, entry_tid=0' -a 'object_number=0, tag_tid=113, entry_tid=0' = 'object_number=0, tag_tid=110, entry_tid=0' 2024-09-16T11:34:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=110, entry_tid=0' '!=' '' 2024-09-16T11:34:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:34:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=110, entry_tid=0' = '' 2024-09-16T11:34:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=110, entry_tid=0' 2024-09-16T11:34:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=113, entry_tid=0' 2024-09-16T11:34:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=113 2024-09-16T11:34:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=110, entry_tid=0' 2024-09-16T11:34:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=110 2024-09-16T11:34:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=113, entry_tid=0' 2024-09-16T11:34:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-09-16T11:34:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=110, entry_tid=0' 2024-09-16T11:34:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 113 = 110 -a 0 -le 0 2024-09-16T11:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:34:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:34:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=113, entry_tid=0' 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:34:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=113, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:34:36.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=113, entry_tid=0' 2024-09-16T11:34:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=113, entry_tid=0' -a 'object_number=0, tag_tid=113, entry_tid=0' = 'object_number=0, tag_tid=113, entry_tid=0' 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:35 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:35' 2024-09-16T11:34:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:34:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:34:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:34:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:35' 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:34:35' 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:35' 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:34:35cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}2024-09-16 11:34:35' 2024-09-16T11:34:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:35' 2024-09-16T11:34:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:34:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:34:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:34:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:35 2024-09-16T11:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:35' 2024-09-16T11:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:34:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}} 2024-09-16T11:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:35' 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:34:35' 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:35' 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":110},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":113}}42884295mirror.0smithi2042024-09-16 11:34:35cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:34:35' 2024-09-16T11:34:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:35' 2024-09-16T11:34:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:34:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:34:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster2 mirror test 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2024-09-16T11:34:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:34:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:34:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:42.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:42.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:40 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:40' 2024-09-16T11:34:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:43.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:40' 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:34:40' 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:40' 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:34:40cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+unknownremote image is not primary2024-09-16 11:34:40' 2024-09-16T11:34:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:40' 2024-09-16T11:34:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:40 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:40' 2024-09-16T11:34:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:40' 2024-09-16T11:34:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:34:40' 2024-09-16T11:34:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:40' 2024-09-16T11:34:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:34:40cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:34:40' 2024-09-16T11:34:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:40' 2024-09-16T11:34:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:34:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2024-09-16T11:34:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:34:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster2 mirror test 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test started 2024-09-16T11:34:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:34:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster2 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:34:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:34:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:34:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:34:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:46.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:34:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:34:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:34:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=111, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=114, entry_tid=0' 2024-09-16T11:34:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:34:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=111, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=111, entry_tid=0' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=114, entry_tid=0' -a 'object_number=0, tag_tid=114, entry_tid=0' = 'object_number=0, tag_tid=111, entry_tid=0' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=111, entry_tid=0' '!=' '' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=0, tag_tid=111, entry_tid=0' = '' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=0, tag_tid=111, entry_tid=0' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=114, entry_tid=0' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=114 2024-09-16T11:34:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=111, entry_tid=0' 2024-09-16T11:34:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:34:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=111 2024-09-16T11:34:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=114, entry_tid=0' 2024-09-16T11:34:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=0 2024-09-16T11:34:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=0, tag_tid=111, entry_tid=0' 2024-09-16T11:34:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:34:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:34:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=0 2024-09-16T11:34:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 114 = 111 -a 0 -le 0 2024-09-16T11:34:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:34:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:34:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster2 mirror test 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:34:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:34:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:34:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster1 mirror test 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '' 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:34:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=114, entry_tid=0' 2024-09-16T11:34:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster1 mirror test 2024-09-16T11:34:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster1 mirror test '..*' 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:34:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:34:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 journal status --image mirror/test 2024-09-16T11:34:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:34:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:34:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:34:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=575d6853-c710-4ab7-a806-f3cf450240e2, commit_position=[positions=[[object_number=0, tag_tid=114, entry_tid=0], [object_number=1, tag_tid=13, entry_tid=101], [object_number=3, tag_tid=13, entry_tid=99], [object_number=2, tag_tid=13, entry_tid=98]]], state=connected] 2024-09-16T11:34:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:34:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=114, entry_tid=0' 2024-09-16T11:34:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=114, entry_tid=0' -a 'object_number=0, tag_tid=114, entry_tid=0' = 'object_number=0, tag_tid=114, entry_tid=0' 2024-09-16T11:34:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped '' '' 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:45 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:45' 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:34:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:34:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:34:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:45' 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped '2024-09-16 11:34:45' 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:45' 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary42884295mirror.0smithi2042024-09-16 11:34:45cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}2024-09-16 11:34:45' 2024-09-16T11:34:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:45' 2024-09-16T11:34:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:34:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:34:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:34:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying '' '' 2024-09-16T11:34:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:45 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:45' 2024-09-16T11:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:34:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:34:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}} 2024-09-16T11:34:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:45' 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying '2024-09-16 11:34:45' 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:45' 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":111},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":114}}44464452mirror.0smithi2042024-09-16 11:34:45cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:34:45' 2024-09-16T11:34:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:45' 2024-09-16T11:34:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:34:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:34:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:34:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:34:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror test 2024-09-16T11:34:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2024-09-16T11:34:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:34:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2024-09-16T11:34:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:34:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:34:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:34:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:34:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:34:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = started 2024-09-16T11:34:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test stopped 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:34:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:34:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:34:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:34:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test 2024-09-16T11:34:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:34:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:34:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:34:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:34:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:34:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:34:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:34:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:34:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:34:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:34:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:34:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:34:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:34:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown '' '' 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:34:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:50 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:51' 2024-09-16T11:34:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:50' 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown '2024-09-16 11:34:50' 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:50' 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary42884295mirror.0smithi2042024-09-16 11:34:50cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedstopped2024-09-16 11:34:51' 2024-09-16T11:34:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:50' 2024-09-16T11:34:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: stopped 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:51 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:50' 2024-09-16T11:34:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:34:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:34:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown '' '' 2024-09-16T11:34:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:34:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:34:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:55 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:34:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:34:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+unknown 2024-09-16T11:34:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: remote image is not primary 2024-09-16T11:34:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:34:55' 2024-09-16T11:34:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T11:34:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+unknown 2024-09-16T11:34:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:34:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: remote image is not primary 2024-09-16T11:34:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:34:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:34:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:34:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:34:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:34:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:55' 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown '2024-09-16 11:34:55' 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+unknown 2024-09-16T11:34:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:34:55' 2024-09-16T11:34:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:34:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:34:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:34:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+unknownremote image is not primary44464452mirror.0smithi2042024-09-16 11:34:55cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+unknownremote image is not primary2024-09-16 11:34:55' 2024-09-16T11:34:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:34:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:34:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:34:55' 2024-09-16T11:34:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:34:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+unknown 2024-09-16T11:34:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+unknown 2024-09-16T11:34:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+unknown 2024-09-16T11:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+unknown 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster2 mirror test 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force= 2024-09-16T11:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2024-09-16T11:34:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:34:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:34:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:34:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:34:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:34:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:34:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:34:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:34:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:34:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:34:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:34:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:34:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:34:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:34:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:34:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:34:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:34:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:34:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:34:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:34:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:34:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:34:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:34:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:34:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:34:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:34:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:34:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:34:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:34:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:34:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:34:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:34:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:34:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:34:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:34:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:34:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:34:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:34:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:34:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:34:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:34:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:34:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:34:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:34:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:34:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:34:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:34:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:34:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:34:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:34:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:34:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:34:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:34:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:34:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:34:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:34:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:35:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:35:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:35:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:35:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:35:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:35:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = stopped 2024-09-16T11:35:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:04.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:35:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:35:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:35:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:35:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-16T11:35:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:35:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:35:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:35:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:35:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:35:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:35:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:35:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=0, tag_tid=118, entry_tid=0' 2024-09-16T11:35:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test.status 2024-09-16T11:35:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:35:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-16T11:35:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:35:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:35:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=0, tag_tid=118, entry_tid=0], [object_number=1, tag_tid=18, entry_tid=101], [object_number=3, tag_tid=18, entry_tid=99], [object_number=2, tag_tid=18, entry_tid=98]]], state=connected] 2024-09-16T11:35:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.status 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=0, tag_tid=118, entry_tid=0' 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=0, tag_tid=118, entry_tid=0' -a 'object_number=0, tag_tid=118, entry_tid=0' = 'object_number=0, tag_tid=118, entry_tid=0' 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped '' '' 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test 2024-09-16T11:35:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:35:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:35:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:05 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:00' 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:35:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:35:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:35:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:35:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:05' 2024-09-16T11:35:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped '2024-09-16 11:35:05' 2024-09-16T11:35:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:05' 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:35:05cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2024-09-16 11:35:00' 2024-09-16T11:35:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:35:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:05' 2024-09-16T11:35:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:35:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:35:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:35:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:35:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:35:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 9b27b81f-bd43-4dc5-bbb4-0c3437048cc7 2024-09-16T11:35:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:35:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2024-09-16T11:35:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:00 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:05' 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:35:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:35:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:35:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}} 2024-09-16T11:35:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:35:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:00' 2024-09-16T11:35:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:35:00' 2024-09-16T11:35:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:00' 2024-09-16T11:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:07.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":115},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42884295mirror.0smithi2042024-09-16 11:35:00cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:05' 2024-09-16T11:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:35:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:00' 2024-09-16T11:35:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:35:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:35:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:35:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: force promote' 2024-09-16T11:35:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:35:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:35:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:35:07 TEST: force promote 2024-09-16T11:35:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:35:07 TEST: force promote 2024-09-16T11:35:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ force_promote_image=test_force_promote 2024-09-16T11:35:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test_force_promote 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_force_promote 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_force_promote 100 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:35:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:35:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:35:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:35:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3125.05 bytes/sec: 12 MiB/s 2024-09-16T11:35:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2024-09-16T11:35:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2024-09-16T11:35:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2024-09-16T11:35:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:35:08.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:35:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:1 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:35:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:35:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:35:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:35:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:35:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:35:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:35:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:1 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:35:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:35:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:35:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:35:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote started 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_force_promote started 2024-09-16T11:35:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:10.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:35:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:35:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_force_promote 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_force_promote 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_force_promote 2024-09-16T11:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_force_promote ']' 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_force_promote' 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:35:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '' 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=101], [object_number=0, tag_tid=1, entry_tid=100], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-09-16T11:35:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*' 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=101], [object_number=0, tag_tid=1, entry_tid=100], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-09-16T11:35:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=101' '!=' '' 2024-09-16T11:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:35:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=101' = '' 2024-09-16T11:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-09-16T11:35:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:35:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-09-16T11:35:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:35:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-09-16T11:35:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:35:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=101 2024-09-16T11:35:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 1 -a 3 -le 101 2024-09-16T11:35:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:35:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_force_promote 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_force_promote ']' 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_force_promote' 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:35:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:35:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:11.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '' 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-09-16T11:35:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*' 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-09-16T11:35:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:35:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=1, tag_tid=1, entry_tid=101' 2024-09-16T11:35:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:35:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-09-16T11:35:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:35:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=2 2024-09-16T11:35:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:35:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-09-16T11:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:35:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-09-16T11:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 -a 3 -le 1 2024-09-16T11:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_force_promote 2024-09-16T11:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_force_promote ']' 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_force_promote' 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:35:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '' 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-09-16T11:35:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*' 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2024-09-16T11:35:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:35:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-09-16T11:35:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_force_promote 2024-09-16T11:35:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_force_promote ']' 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_force_promote' 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:35:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:35:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:35:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:35:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_force_promote 2024-09-16T11:35:12.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '' 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_force_promote 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_force_promote '..*' 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_force_promote 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2024-09-16T11:35:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_force_promote.status 2024-09-16T11:35:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position '' 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2024-09-16T11:35:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-09-16T11:35:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 32e4da04-37a1-486e-b964-d0bce8e4d4cd 2024-09-16T11:35:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:35:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:35:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:10 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:10' 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:35:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:35:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:35:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:35:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:35:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:10' 2024-09-16T11:35:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+replaying '2024-09-16 11:35:10' 2024-09-16T11:35:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:10' 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42884295mirror.0smithi2042024-09-16 11:35:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:10test_force_promote32e4da04-37a1-486e-b964-d0bce8e4d4cdup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42944297mirror.1smithi2042024-09-16 11:35:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:10' 2024-09-16T11:35:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-09-16T11:35:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:10' 2024-09-16T11:35:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-09-16T11:35:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:35:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:35:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 32e4da04-37a1-486e-b964-d0bce8e4d4cd 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:10 2024-09-16T11:35:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:35:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:35:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:35:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:10' 2024-09-16T11:35:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:35:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:35:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:35:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:35:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:35:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:10' 2024-09-16T11:35:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2024-09-16 11:35:10' 2024-09-16T11:35:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:10' 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK244464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:35:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2024-09-16 11:35:10test_force_promote32e4da04-37a1-486e-b964-d0bce8e4d4cdup+stoppedlocal image is primary44534459mirror.1smithi2042024-09-16 11:35:10cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}2024-09-16 11:35:10' 2024-09-16T11:35:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-09-16T11:35:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:10' 2024-09-16T11:35:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-09-16T11:35:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:35:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:35:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror test_force_promote --force 2024-09-16T11:35:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force=--force 2024-09-16T11:35:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote --force 2024-09-16T11:35:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote stopped 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_force_promote stopped 2024-09-16T11:35:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:35:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:16.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:35:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:35:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:35:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2024-09-16T11:35:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster2 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster2 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:35:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:0 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:35:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:35:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:35:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:35:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:35:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2024-09-16T11:35:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2024-09-16T11:35:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-09-16T11:35:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster2:1 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster2 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster2 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:35:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:35:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2024-09-16T11:35:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:35:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:35:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+stopped 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+stopped '' '' 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_force_promote 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 32e4da04-37a1-486e-b964-d0bce8e4d4cd 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:15 2024-09-16T11:35:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:15' 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:35:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:35:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:35:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:35:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:15' 2024-09-16T11:35:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+stopped '2024-09-16 11:35:15' 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:15' 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK1142884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42884295mirror.0smithi2042024-09-16 11:35:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:15test_force_promote32e4da04-37a1-486e-b964-d0bce8e4d4cdup+stoppedlocal image is primary42944297mirror.1smithi2042024-09-16 11:35:15cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:15' 2024-09-16T11:35:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-09-16T11:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:15' 2024-09-16T11:35:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-09-16T11:35:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:35:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:35:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped '' '' 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror image status mirror/test_force_promote 2024-09-16T11:35:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_force_promote: 2024-09-16T11:35:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 32e4da04-37a1-486e-b964-d0bce8e4d4cd 2024-09-16T11:35:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:15 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster1 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:15' 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:35:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:35:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:35:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:35:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:35:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:15' 2024-09-16T11:35:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped '2024-09-16 11:35:15' 2024-09-16T11:35:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:15' 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK1144464452mirror.0smithi20417.2.7-1643-gd75f2382trueOK44534459mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+stoppedlocal image is primary44464452mirror.0smithi2042024-09-16 11:35:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}2024-09-16 11:35:10test_force_promote32e4da04-37a1-486e-b964-d0bce8e4d4cdup+stoppedlocal image is primary44534459mirror.1smithi2042024-09-16 11:35:15cluster188cac606-ed81-47bf-b993-eb31e1d47ba4up+stoppedlocal image is primary2024-09-16 11:35:15' 2024-09-16T11:35:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/last_update' 2024-09-16T11:35:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:15' 2024-09-16T11:35:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_force_promote'\'']/state' 2024-09-16T11:35:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:35:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:35:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:35:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:35:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster1 mirror test_force_promote 100 2024-09-16T11:35:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:35:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:35:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:35:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:35:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3703.76 bytes/sec: 14 MiB/s 2024-09-16T11:35:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_force_promote 100 2024-09-16T11:35:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:35:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:35:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:35:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:35:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2564.14 bytes/sec: 10 MiB/s 2024-09-16T11:35:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster1 mirror test_force_promote 2024-09-16T11:35:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster1 mirror test_force_promote 2024-09-16T11:35:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_force_promote 2024-09-16T11:35:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 rm mirror/test_force_promote 2024-09-16T11:35:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_force_promote 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_force_promote 2024-09-16T11:35:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_force_promote 2024-09-16T11:35:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_force_promote 2024-09-16T11:35:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_force_promote 2024-09-16T11:35:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:35:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: cloned images' 2024-09-16T11:35:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:35:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:35:20 TEST: cloned images 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:35:20 TEST: cloned images 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - default' 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:35:20 - default 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:35:20 - default 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ parent_image=test_parent 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ parent_snap=snap 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror_parent test_parent 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:35:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror_parent/test_parent 2024-09-16T11:35:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror_parent test_parent 100 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:35:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror_parent/test_parent --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:35:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:35:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:35:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2777.82 bytes/sec: 11 MiB/s 2024-09-16T11:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap 2024-09-16T11:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-09-16T11:35:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap 2024-09-16T11:35:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:35:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror_parent test_parent snap 2024-09-16T11:35:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-09-16T11:35:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror_parent/test_parent@snap 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image=test_clone 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone 2024-09-16T11:35:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-09-16T11:35:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone --image-feature layering,exclusive-lock,journaling 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_clone 100 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:35:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:35:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:35:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_clone --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:35:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:35:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:35:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 518.143 bytes/sec: 2.0 MiB/s 2024-09-16T11:35:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal 2024-09-16T11:35:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=journal 2024-09-16T11:35:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal 2024-09-16T11:35:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2024-09-16T11:35:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror_parent/test_parent 2024-09-16T11:35:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test_parent': 2024-09-16T11:35:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 15ed7747c5d6 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.15ed7747c5d6 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling, operations 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: clone-parent 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:35:20 2024 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:35:20 2024 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:35:20 2024 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 15ed7747c5d6 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:35:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:35:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: fc2056a6-05ad-440e-92b8-6a62d49c6f35 2024-09-16T11:35:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror_parent test_parent 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror_parent test_parent started 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror_parent test_parent started 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-09-16T11:35:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-09-16T11:35:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:35:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-09-16T11:35:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-09-16T11:35:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:35:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-09-16T11:35:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2024-09-16T11:35:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 4 2024-09-16T11:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2024-09-16T11:35:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:35:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:35:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror_parent test_parent 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror_parent test_parent 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:35:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror_parent test_parent 2024-09-16T11:35:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_parent ']' 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror_parent/test_parent' 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:30.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent 2024-09-16T11:35:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror_parent test_parent 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror_parent test_parent '' 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2024-09-16T11:35:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-09-16T11:35:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=bc3b9b2a-4979-4286-8a03-ed9d7a0879cd, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-09-16T11:35:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=4, entry_tid=3' 2024-09-16T11:35:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror_parent test_parent 2024-09-16T11:35:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror_parent test_parent '..*' 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror_parent 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_parent 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2024-09-16T11:35:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-09-16T11:35:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=bc3b9b2a-4979-4286-8a03-ed9d7a0879cd, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2024-09-16T11:35:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.status 2024-09-16T11:35:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=4, entry_tid=3' 2024-09-16T11:35:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=4, entry_tid=3' -a 'object_number=3, tag_tid=4, entry_tid=3' = 'object_number=3, tag_tid=4, entry_tid=3' 2024-09-16T11:35:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position 2024-09-16T11:35:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:35:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position '' 2024-09-16T11:35:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror_parent/test_parent 2024-09-16T11:35:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_parent: 2024-09-16T11:35:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: fc2056a6-05ad-440e-92b8-6a62d49c6f35 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:25 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:31' 2024-09-16T11:35:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:35:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:35:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:35:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-09-16T11:35:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:25' 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror_parent test_parent up+replaying '2024-09-16 11:35:25' 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:25' 2024-09-16T11:35:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror_parent --verbose --format xml 2024-09-16T11:35:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142884300mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944303mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest_parentfc2056a6-05ad-440e-92b8-6a62d49c6f35up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}42884300mirror.0smithi2042024-09-16 11:35:25cluster27f7cfcdb-fedc-47bc-855b-35e3a197eb01up+stoppedlocal image is primary2024-09-16 11:35:31' 2024-09-16T11:35:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/last_update' 2024-09-16T11:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:25' 2024-09-16T11:35:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_parent'\'']/state' 2024-09-16T11:35:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:35:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:35:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror_parent test_parent 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror_parent-test_parent.export 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror_parent-test_parent.export 2024-09-16T11:35:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.export 2024-09-16T11:35:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror_parent/test_parent /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.export 2024-09-16T11:35:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:35:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror_parent/test_parent /tmp/tmp.MThcdnqr7J/cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:35:31.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_parent-test_parent.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_parent-test_parent.export 2024-09-16T11:35:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone started 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:31.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_clone started 2024-09-16T11:35:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone 2024-09-16T11:35:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2024-09-16T11:35:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:35:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:35:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:35:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:35:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_clone 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_clone 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:35:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:35:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:35:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:35:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:35:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_clone 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_clone ']' 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_clone' 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_clone 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:35:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_clone 2024-09-16T11:35:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:35:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_clone 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_clone 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_clone '' 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-09-16T11:35:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:35:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_clone.status 2024-09-16T11:35:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_clone.status 2024-09-16T11:35:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.status 2024-09-16T11:35:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone 2024-09-16T11:35:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.status 2024-09-16T11:35:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.status 2024-09-16T11:35:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_clone 2024-09-16T11:35:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_clone '..*' 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_clone.status 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_clone.status 2024-09-16T11:35:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.status 2024-09-16T11:35:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_clone 2024-09-16T11:35:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.status 2024-09-16T11:35:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:35:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:35:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:35:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:35:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.status 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position '' 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:35:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_clone 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_clone: 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: c824bd51-1c0f-409b-8ad2-92f1ccbeef9e 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:27 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:35:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:35:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:35:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:35:30' 2024-09-16T11:35:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:35:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:35:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:35:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:35:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:35:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:35:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:35:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:35:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:35:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:27' 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_clone up+replaying '2024-09-16 11:35:27' 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:35:27' 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:35:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:35:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":6.0,"entries_behind_primary":0,"entries_per_second":0.25,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42884295mirror.0smithi2042024-09-16 11:35:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:30test_clonec824bd51-1c0f-409b-8ad2-92f1ccbeef9eup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42884295mirror.0smithi2042024-09-16 11:35:27cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:35:30' 2024-09-16T11:35:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:35:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/last_update' 2024-09-16T11:35:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:35:27' 2024-09-16T11:35:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_clone'\'']/state' 2024-09-16T11:35:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:35:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:35:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:35:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:35:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_clone 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_clone.export 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_clone.export 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.export 2024-09-16T11:35:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_clone.export 2024-09-16T11:35:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_clone.export 2024-09-16T11:35:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:35:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test_clone.export 2024-09-16T11:35:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_clone.export 2024-09-16T11:35:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_clone /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.export 2024-09-16T11:35:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:35:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_clone /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_clone.export 2024-09-16T11:35:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:35:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_clone.export 2024-09-16T11:35:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_clone.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_clone.export 2024-09-16T11:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_clone 2024-09-16T11:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_clone 2024-09-16T11:35:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone 2024-09-16T11:35:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone 2024-09-16T11:35:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone 2024-09-16T11:35:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:35:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - clone v1' 2024-09-16T11:35:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:35:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:35:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:35:35 - clone v1 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:35:35 - clone v1 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster1 mirror_parent test_parent snap mirror test_clone1 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone1 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-09-16T11:35:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap mirror/test_clone1 --image-feature layering,exclusive-lock,journaling 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone_v1 --rbd-default-clone-format 1 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone_v1 2024-09-16T11:35:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-09-16T11:35:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone_v1 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 1 2024-09-16T11:35:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster2 mirror test_clone_v1 2024-09-16T11:35:35.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v1 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v1 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-09-16T11:35:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-09-16T11:35:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-09-16T11:35:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-09-16T11:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 2024-09-16T11:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v1 2024-09-16T11:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v1 started 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_clone_v1 started 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-09-16T11:35:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-09-16T11:35:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:35:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-09-16T11:35:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-09-16T11:35:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:35:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2024-09-16T11:35:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2024-09-16T11:35:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster1 mirror test_clone_v1 2024-09-16T11:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v1 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v1 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-09-16T11:35:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-09-16T11:35:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-09-16T11:35:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-09-16T11:35:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 2024-09-16T11:35:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_clone_v1 2024-09-16T11:35:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-09-16T11:35:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_clone_v1 2024-09-16T11:35:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v1 2024-09-16T11:35:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v1 2024-09-16T11:35:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v1 2024-09-16T11:35:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:35:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster1 mirror test_clone1 2024-09-16T11:35:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone1 2024-09-16T11:35:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster1 mirror test_clone1 2024-09-16T11:35:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone1 2024-09-16T11:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone1 2024-09-16T11:35:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone1 2024-09-16T11:35:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot_retry cluster2 mirror_parent test_parent snap 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror_parent test_parent snap 2024-09-16T11:35:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-09-16T11:35:40.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror_parent/test_parent@snap 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap 2024-09-16T11:35:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap 2024-09-16T11:35:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:35:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - clone v2' 2024-09-16T11:35:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:35:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:35:40 - clone v2 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:35:40 - clone v2 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ parent_snap=snap_v2 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-09-16T11:35:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2024-09-16T11:35:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap_v2 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone_v2 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-09-16T11:35:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster2 mirror test_clone_v2 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/test_clone_v2 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-09-16T11:35:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-09-16T11:35:41.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-09-16T11:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 2024-09-16T11:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v2 2024-09-16T11:35:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v2 started 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_clone_v2 started 2024-09-16T11:35:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:35:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:35:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:35:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:35:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:35:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:35:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:35:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:35:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:42.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:42.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:35:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:44.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:35:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:35:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 4 2024-09-16T11:35:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:35:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:35:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:35:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:35:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2024-09-16T11:35:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:35:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:35:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:35:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:35:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:35:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:35:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:35:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:35:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:35:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2024-09-16T11:35:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:35:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:35:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:35:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:35:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_clone_format cluster1 mirror test_clone_v2 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/test_clone_v2 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk 'BEGIN { 2024-09-16T11:35:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 1 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: $1 == "parent:" { 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: parent = $2 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: /op_features: .*clone-child/ { 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: format = 2 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: } 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: END { 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: if (!parent) exit 1 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: print format 2024-09-16T11:35:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: }' 2024-09-16T11:35:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 2 2024-09-16T11:35:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-09-16T11:35:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2024-09-16T11:35:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:35:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:35:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2024-09-16T11:35:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:35:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 8 7bf3f4ec-7334-4fb0-a246-89a16d375da8 128 MiB Mon Sep 16 11:35:41 2024 trash (snap_v2) 2024-09-16T11:35:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:35:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-09-16T11:35:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:35:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:35:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:35:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:35:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:35:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-09-16T11:35:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:35:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 8 ec9eb7a8-47b6-49d0-aa8d-3fe484e1b0f8 128 MiB Mon Sep 16 11:35:45 2024 trash (snap_v2) 2024-09-16T11:35:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_clone_v2 2024-09-16T11:35:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_clone_v2 2024-09-16T11:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v2 2024-09-16T11:35:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v2 2024-09-16T11:35:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test_clone_v2 deleted 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:35:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_clone_v2 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2024-09-16T11:35:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:35:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2024-09-16T11:35:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:35:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:35:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test_clone_v2 deleted '' 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_clone_v2 2024-09-16T11:35:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:35:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:35:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_clone_v2 2024-09-16T11:35:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_clone_v2 2024-09-16T11:35:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:35:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:35:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2024-09-16T11:35:54.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:35:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:35:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:35:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-09-16T11:35:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:35:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - clone v2 non-primary' 2024-09-16T11:35:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:35:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:35:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:35:55 - clone v2 non-primary 2024-09-16T11:35:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:35:55 - clone v2 non-primary 2024-09-16T11:35:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-09-16T11:35:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2024-09-16T11:35:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:35:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:35:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:35:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:35:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:35:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:35:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2024-09-16T11:35:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:35:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:35:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:35:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:35:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test_parent': 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 1 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10cc9a37b633 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10cc9a37b633 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:35:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:35:23 2024 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:35:23 2024 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:35:23 2024 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10cc9a37b633 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: fc2056a6-05ad-440e-92b8-6a62d49c6f35 2024-09-16T11:35:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ clone_image cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_pool=mirror_parent 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_image=test_parent 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local parent_snap=snap_v2 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_pool=mirror 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local clone_image=test_clone_v2 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 6 2024-09-16T11:35:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2024-09-16T11:35:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:35:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:35:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:35:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:35:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_v2 2024-09-16T11:35:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2024-09-16T11:36:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:36:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2024-09-16T11:36:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:36:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:36:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:36:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:36:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:36:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:36:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-09-16T11:36:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:36:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: 10 b330a6ec-0d21-4c48-ad73-83bd46f58e4c 128 MiB Mon Sep 16 11:35:58 2024 trash (snap_v2) 2024-09-16T11:36:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster1 mirror test_clone_v2 2024-09-16T11:36:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster1 mirror test_clone_v2 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_clone_v2 2024-09-16T11:36:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone_v2 2024-09-16T11:36:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone_v2 2024-09-16T11:36:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:36:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:36:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_v2 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2024-09-16T11:36:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -F ' trash (snap_v2)' 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror_parent test_parent 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:36:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror_parent test_parent 2024-09-16T11:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror_parent 2024-09-16T11:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_parent 2024-09-16T11:36:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror_parent/test_parent 2024-09-16T11:36:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror_parent/test_parent 2024-09-16T11:36:03.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: data pool' 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:36:03 TEST: data pool 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:36:03 TEST: data pool 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ dp_image=test_data_pool 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test_data_pool 128 --data-pool mirror_parent 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/test_data_pool 2024-09-16T11:36:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster2 mirror test_data_pool 2024-09-16T11:36:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/test_data_pool 2024-09-16T11:36:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test_data_pool 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test_data_pool started 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:36:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test_data_pool started 2024-09-16T11:36:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:36:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:04.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:36:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2024-09-16T11:36:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:36:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2024-09-16T11:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:36:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:36:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:36:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2024-09-16T11:36:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2024-09-16T11:36:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:36:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:36:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:36:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:36:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2024-09-16T11:36:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:07.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:36:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2024-09-16T11:36:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:36:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:36:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:36:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:36:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster1 mirror test_data_pool 2024-09-16T11:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/test_data_pool 2024-09-16T11:36:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test_data_pool snap1 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap1 2024-09-16T11:36:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_data_pool 100 2024-09-16T11:36:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3333.38 bytes/sec: 13 MiB/s 2024-09-16T11:36:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test_data_pool snap2 2024-09-16T11:36:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap2 2024-09-16T11:36:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_data_pool 100 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 689.666 bytes/sec: 2.7 MiB/s 2024-09-16T11:36:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_data_pool 2024-09-16T11:36:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_data_pool 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:36:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:36:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:36:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:36:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-09-16T11:36:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=5 2024-09-16T11:36:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-09-16T11:36:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:36:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-09-16T11:36:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 5 = 1 -a 101 -le 3 2024-09-16T11:36:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:36:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-09-16T11:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2024-09-16T11:36:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2024-09-16T11:36:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=5 2024-09-16T11:36:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=2 2024-09-16T11:36:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:36:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-09-16T11:36:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 5 = 2 -a 101 -le 1 2024-09-16T11:36:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:36:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:36:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-09-16T11:36:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:11.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2024-09-16T11:36:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-09-16T11:36:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2024-09-16T11:36:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=2, entry_tid=1' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-09-16T11:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-09-16T11:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:12.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:12.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=3, entry_tid=101' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' '!=' 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=3, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=1' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=3, entry_tid=101' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=5 2024-09-16T11:36:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101' 2024-09-16T11:36:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=3 2024-09-16T11:36:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:36:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=3, entry_tid=101' 2024-09-16T11:36:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=101 2024-09-16T11:36:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 5 = 3 -a 101 -le 101 2024-09-16T11:36:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:36:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:36:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_data_pool 2024-09-16T11:36:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_data_pool' 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2024-09-16T11:36:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_data_pool 2024-09-16T11:36:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '' 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_data_pool 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_data_pool '..*' 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2024-09-16T11:36:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2024-09-16T11:36:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.status 2024-09-16T11:36:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=5, entry_tid=101' -a 'object_number=1, tag_tid=5, entry_tid=101' = 'object_number=1, tag_tid=5, entry_tid=101' 2024-09-16T11:36:12.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position '' 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:36:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_data_pool 2024-09-16T11:36:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_data_pool: 2024-09-16T11:36:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 3e7c62ff-65b8-4431-bc48-2bc4310f0c36 2024-09-16T11:36:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:36:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":100.25,"entries_behind_primary":204,"entries_per_second":1.25,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"seconds_until_synced":163} 2024-09-16T11:36:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:36:10 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:36:10' 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:36:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:36:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:36:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":100.25,"entries_behind_primary":204,"entries_per_second":1.25,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"seconds_until_synced":163} 2024-09-16T11:36:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:36:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:36:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:36:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:36:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:36:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:36:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:36:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:36:10' 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_data_pool up+replaying '2024-09-16 11:36:10' 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:36:10' 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:36:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:36:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK242884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42884295mirror.0smithi2042024-09-16 11:36:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:36:10test_data_pool3e7c62ff-65b8-4431-bc48-2bc4310f0c36up+replayingreplaying, {"bytes_per_second":100.25,"entries_behind_primary":204,"entries_per_second":1.25,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"seconds_until_synced":163}42944297mirror.1smithi2042024-09-16 11:36:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:36:10' 2024-09-16T11:36:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:36:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2024-09-16T11:36:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:36:10' 2024-09-16T11:36:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2024-09-16T11:36:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:36:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:36:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:36:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:36:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_data_pool@snap1 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool@snap1 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap1 /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap1.export 2024-09-16T11:36:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap1 /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap1.export 2024-09-16T11:36:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_data_pool@snap2 2024-09-16T11:36:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:14.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool@snap2 2024-09-16T11:36:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:36:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:14.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap2 /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap2 /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool@snap2.export 2024-09-16T11:36:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test_data_pool 2024-09-16T11:36:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:36:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test_data_pool.export 2024-09-16T11:36:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test_data_pool.export 2024-09-16T11:36:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.export 2024-09-16T11:36:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test_data_pool.export 2024-09-16T11:36:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test_data_pool.export 2024-09-16T11:36:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool.export 2024-09-16T11:36:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool.export 2024-09-16T11:36:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.export 2024-09-16T11:36:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool.export 2024-09-16T11:36:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool.export 2024-09-16T11:36:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_data_pool.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test_data_pool.export 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_data_pool 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:36:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_data_pool 2024-09-16T11:36:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:36:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_data_pool 2024-09-16T11:36:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-16T11:36:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_data_pool 2024-09-16T11:36:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:36:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: disable mirroring / delete non-primary image' 2024-09-16T11:36:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:36:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:36:16 TEST: disable mirroring / delete non-primary image 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:36:16 TEST: disable mirroring / delete non-primary image 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image2=test2 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image3=test3 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image4=test4 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image5=test5 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test2 2024-09-16T11:36:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:36:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test2 2024-09-16T11:36:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test2 100 2024-09-16T11:36:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3125.05 bytes/sec: 12 MiB/s 2024-09-16T11:36:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test2 snap1 2024-09-16T11:36:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap1 2024-09-16T11:36:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test2 snap2 2024-09-16T11:36:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap2 2024-09-16T11:36:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test2 = test4 ']' 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test2 = test5 ']' 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test2 100 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 1052.65 bytes/sec: 4.1 MiB/s 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test2 present 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10c7b725cf76 2024-09-16T11:36:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present 10c7b725cf76 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c7b725cf76 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c7b725cf76 2024-09-16T11:36:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c7b725cf76 2024-09-16T11:36:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c7b725cf76 -o 10c7b725cf76 = 10c7b725cf76 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-09-16T11:36:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:36:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:36:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:36:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-09-16T11:36:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test2': 2024-09-16T11:36:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c7b725cf76 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c7b725cf76 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:19 2024 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:19 2024 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:19 2024 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c7b725cf76 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:36:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: fa3fa982-7f66-4262-b061-6fa0c3f143bf 2024-09-16T11:36:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test3 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:36:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:36:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test3 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test3 100 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3030.35 bytes/sec: 12 MiB/s 2024-09-16T11:36:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test3 snap1 2024-09-16T11:36:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap1 2024-09-16T11:36:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test3 snap2 2024-09-16T11:36:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap2 2024-09-16T11:36:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test3 = test4 ']' 2024-09-16T11:36:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test3 = test5 ']' 2024-09-16T11:36:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test3 100 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 512.828 bytes/sec: 2.0 MiB/s 2024-09-16T11:36:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test3 present 2024-09-16T11:36:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-09-16T11:36:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10c9caaa710e 2024-09-16T11:36:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test3 present 10c9caaa710e 2024-09-16T11:36:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:26.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c9caaa710e 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-09-16T11:36:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c9caaa710e 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c9caaa710e 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c9caaa710e -o 10c9caaa710e = 10c9caaa710e 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test3 snap2 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-09-16T11:36:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2024-09-16T11:36:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test3': 2024-09-16T11:36:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c9caaa710e 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c9caaa710e 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:24 2024 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:24 2024 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:24 2024 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c9caaa710e 2024-09-16T11:36:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:36:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:36:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9179ab35-2e23-43c6-b2d9-302de2b6a07f 2024-09-16T11:36:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:36:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test4 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:36:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test4 2024-09-16T11:36:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test4 100 2024-09-16T11:36:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2857.19 bytes/sec: 11 MiB/s 2024-09-16T11:36:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test4 snap1 2024-09-16T11:36:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap1 2024-09-16T11:36:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test4 snap2 2024-09-16T11:36:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap2 2024-09-16T11:36:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test4 = test4 ']' 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test4 snap1 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap1 2024-09-16T11:36:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test4 snap2 2024-09-16T11:36:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap2 2024-09-16T11:36:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test4 100 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 699.312 bytes/sec: 2.7 MiB/s 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test4 present 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:29.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-09-16T11:36:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c91c3baf9d 2024-09-16T11:36:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c91c3baf9d 2024-09-16T11:36:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c91c3baf9d 2024-09-16T11:36:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2024-09-16T11:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test4': 2024-09-16T11:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-09-16T11:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c91c3baf9d 2024-09-16T11:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c91c3baf9d 2024-09-16T11:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:29 2024 2024-09-16T11:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:29 2024 2024-09-16T11:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:29 2024 2024-09-16T11:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c91c3baf9d 2024-09-16T11:36:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:36:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:36:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 3d4d7e70-5fae-45b2-a09b-0c897965ed35 2024-09-16T11:36:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:36:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test5 2024-09-16T11:36:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:36:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:36:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test5 2024-09-16T11:36:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test5 100 2024-09-16T11:36:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2702.74 bytes/sec: 11 MiB/s 2024-09-16T11:36:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test5 snap1 2024-09-16T11:36:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap1 2024-09-16T11:36:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test5 snap2 2024-09-16T11:36:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap2 2024-09-16T11:36:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test5 = test4 ']' 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' test5 = test5 ']' 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test5 snap1 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap1 2024-09-16T11:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ protect_snapshot cluster2 mirror test5 snap2 2024-09-16T11:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap2 2024-09-16T11:36:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test5 100 2024-09-16T11:36:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:36:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:36:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:36:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:36:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 518.143 bytes/sec: 2.0 MiB/s 2024-09-16T11:36:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test5 present 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-09-16T11:36:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-09-16T11:36:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-09-16T11:36:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test5 present '' 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-09-16T11:36:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-09-16T11:36:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-09-16T11:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test5 present '' 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-09-16T11:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-09-16T11:36:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c7255e273e 2024-09-16T11:36:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c7255e273e 2024-09-16T11:36:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c7255e273e 2024-09-16T11:36:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test5 snap2 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:35.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test5': 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 2 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c7255e273e 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c7255e273e 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:36:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:33 2024 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:33 2024 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:33 2024 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c7255e273e 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: ed157c85-f14f-455e-b217-1863b41579ea 2024-09-16T11:36:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:36:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-16T11:36:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=image 2024-09-16T11:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-16T11:36:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from pool to image 2024-09-16T11:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-09-16T11:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror test2 2024-09-16T11:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test2 2024-09-16T11:36:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-09-16T11:36:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-09-16T11:36:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror test4 2024-09-16T11:36:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test4 2024-09-16T11:36:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-09-16T11:36:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test5 snap1 2024-09-16T11:36:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap1 2024-09-16T11:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test5 snap2 2024-09-16T11:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap2 2024-09-16T11:36:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image3} ${image5} 2024-09-16T11:36:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test3 snap1 2024-09-16T11:36:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap1 2024-09-16T11:36:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:36:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test3 snap2 2024-09-16T11:36:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap2 2024-09-16T11:36:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:36:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test3 2024-09-16T11:36:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test3 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test3 2024-09-16T11:36:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test3 2024-09-16T11:36:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:36:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image3} ${image5} 2024-09-16T11:36:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test5 snap1 2024-09-16T11:36:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:36:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap1 2024-09-16T11:36:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:36:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_snapshot cluster2 mirror test5 snap2 2024-09-16T11:36:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:36:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap2 2024-09-16T11:36:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing snap: 100% complete...done. 2024-09-16T11:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test5 2024-09-16T11:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test5 2024-09-16T11:36:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test5 2024-09-16T11:36:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test5 2024-09-16T11:36:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:36:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test2 deleted 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10c7b725cf76 2024-09-16T11:36:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 deleted 10c7b725cf76 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c7b725cf76 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c7b725cf76 2024-09-16T11:36:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c7b725cf76 2024-09-16T11:36:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c7b725cf76 -o 10c7b725cf76 = 10c7b725cf76 2024-09-16T11:36:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:36:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 deleted 10c7b725cf76 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c7b725cf76 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test3 deleted 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-09-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test3: (2) No such file or directory 2024-09-16T11:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test3 deleted '' 2024-09-16T11:36:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test3 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test3 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test3 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test3 2024-09-16T11:36:41.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test3: (2) No such file or directory 2024-09-16T11:36:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:36:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test4 deleted 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:41.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 deleted '' 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image3} ${image4} ${image5} 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test5 deleted 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-09-16T11:36:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-09-16T11:36:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test5 deleted '' 2024-09-16T11:36:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test5 2024-09-16T11:36:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test5 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test5 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test5 2024-09-16T11:36:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test5: (2) No such file or directory 2024-09-16T11:36:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:36:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2024-09-16T11:36:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=pool 2024-09-16T11:36:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2024-09-16T11:36:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from image to pool 2024-09-16T11:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-09-16T11:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_journaling cluster2 mirror test2 2024-09-16T11:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 feature enable mirror/test2 journaling 2024-09-16T11:36:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test2 present 2024-09-16T11:36:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-09-16T11:36:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-09-16T11:36:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-09-16T11:36:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-09-16T11:36:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:36:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-09-16T11:36:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test2: (2) No such file or directory 2024-09-16T11:36:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:36:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test2 present '' 2024-09-16T11:36:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test2 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test2 2024-09-16T11:36:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c9a3290aa0 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c9a3290aa0 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c9a3290aa0 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-09-16T11:36:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:36:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:36:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:36:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2024-09-16T11:36:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test2': 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 3 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c9a3290aa0 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c9a3290aa0 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:48 2024 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:48 2024 2024-09-16T11:36:52.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:48 2024 2024-09-16T11:36:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:36:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c9a3290aa0 2024-09-16T11:36:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:36:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:36:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 7f673670-80a6-4986-89e8-661d7e93cb9e 2024-09-16T11:36:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:36:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test2 2024-09-16T11:36:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test2 started 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test2 started 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:36:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test2 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:36:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:36:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test2 2024-09-16T11:36:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:36:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:36:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test2 2024-09-16T11:36:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test2'] 2024-09-16T11:36:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:36:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test2 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:36:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:36:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:36:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:36:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test2 2024-09-16T11:36:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:36:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:36:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:36:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:36:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:36:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:36:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test2 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test2 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:36:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:36:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:36:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:36:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test2 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test2 ']' 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test2' 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2024-09-16T11:36:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test2'] 2024-09-16T11:36:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test2 2024-09-16T11:36:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test2 2024-09-16T11:36:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test2 2024-09-16T11:36:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '' 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-09-16T11:36:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-09-16T11:36:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test2 2024-09-16T11:36:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '..*' 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-09-16T11:36:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-09-16T11:36:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2024-09-16T11:36:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=1, tag_tid=1, entry_tid=1' 2024-09-16T11:36:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' '!=' '' 2024-09-16T11:36:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:36:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=1, tag_tid=1, entry_tid=1' = '' 2024-09-16T11:36:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=1, tag_tid=1, entry_tid=1' 2024-09-16T11:36:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=1 2024-09-16T11:36:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2024-09-16T11:36:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:36:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-09-16T11:36:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-09-16T11:36:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=1, entry_tid=1' 2024-09-16T11:36:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:36:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:36:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-09-16T11:36:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 1 -a 3 -le 1 2024-09-16T11:36:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:36:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:36:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test2 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test2 ']' 2024-09-16T11:36:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test2' 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:36:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:36:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2024-09-16T11:36:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:36:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:36:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:36:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:36:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:36:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2024-09-16T11:36:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test2'] 2024-09-16T11:36:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:36:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test2 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:36:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:36:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:36:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:36:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:36:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test2 2024-09-16T11:36:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test2 2024-09-16T11:36:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '' 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-09-16T11:36:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-09-16T11:36:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test2 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test2 '..*' 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test2 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test2.status 2024-09-16T11:36:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test2.status 2024-09-16T11:36:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test2 2024-09-16T11:36:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:36:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:36:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:36:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:36:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.status 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test2 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test2.export 2024-09-16T11:36:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test2.export 2024-09-16T11:36:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.export 2024-09-16T11:36:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test2.export 2024-09-16T11:36:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test2.export 2024-09-16T11:36:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:36:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test2.export 2024-09-16T11:36:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test2.export 2024-09-16T11:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test2 /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.export 2024-09-16T11:36:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test2 /tmp/tmp.MThcdnqr7J/cluster1-mirror-test2.export 2024-09-16T11:36:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:36:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test2.export 2024-09-16T11:36:54.833 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test2.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test2.export 2024-09-16T11:36:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-09-16T11:36:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_journaling cluster2 mirror test4 2024-09-16T11:36:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:36:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 feature enable mirror/test4 journaling 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test4 present 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:36:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:36:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-09-16T11:36:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test4: (2) No such file or directory 2024-09-16T11:36:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:36:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:36:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:36:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:36:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test4 present '' 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:36:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test4 2024-09-16T11:36:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:36:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:36:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:36:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test4 2024-09-16T11:36:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:36:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c77486a5a5 2024-09-16T11:36:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c77486a5a5 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c77486a5a5 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap2 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:36:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2024-09-16T11:36:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:36:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:36:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:36:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2024-09-16T11:37:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test4': 2024-09-16T11:37:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:37:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:37:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 3 2024-09-16T11:37:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c77486a5a5 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c77486a5a5 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:56 2024 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:56 2024 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:56 2024 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: True 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c77486a5a5 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 9ccd4b16-1909-4bc0-84ae-628353f95e57 2024-09-16T11:37:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:37:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test4 2024-09-16T11:37:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test4 started 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:37:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test4 started 2024-09-16T11:37:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:37:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:37:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test4 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:02.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:37:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:37:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test4 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:37:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:37:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:37:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:37:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:37:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:37:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:37:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4 2024-09-16T11:37:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:37:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:37:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:37:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:37:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:37:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:37:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:37:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test4 2024-09-16T11:37:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test4 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:37:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test4 2024-09-16T11:37:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:37:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test4 ']' 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test4' 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4 2024-09-16T11:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:37:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test4 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test4 '' 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:37:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:37:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test4.status 2024-09-16T11:37:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test4.status 2024-09-16T11:37:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.status 2024-09-16T11:37:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2024-09-16T11:37:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.status 2024-09-16T11:37:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:37:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:37:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:37:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:37:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:37:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.status 2024-09-16T11:37:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:37:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test4 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test4 '..*' 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test4 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test4.status 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test4.status 2024-09-16T11:37:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.status 2024-09-16T11:37:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test4 2024-09-16T11:37:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.status 2024-09-16T11:37:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:37:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:37:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:37:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:37:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:37:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.status 2024-09-16T11:37:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:37:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=1, entry_tid=3' -a 'object_number=3, tag_tid=1, entry_tid=3' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test4 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test4.export 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test4.export 2024-09-16T11:37:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.export 2024-09-16T11:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test4.export 2024-09-16T11:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test4.export 2024-09-16T11:37:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test4.export 2024-09-16T11:37:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test4.export 2024-09-16T11:37:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test4 /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.export 2024-09-16T11:37:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test4 /tmp/tmp.MThcdnqr7J/cluster1-mirror-test4.export 2024-09-16T11:37:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:37:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test4.export 2024-09-16T11:37:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test4.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test4.export 2024-09-16T11:37:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: remove mirroring pool' 2024-09-16T11:37:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:37:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:37:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:37:03 TEST: remove mirroring pool 2024-09-16T11:37:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:37:03 TEST: remove mirroring pool 2024-09-16T11:37:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pool=pool_to_remove 2024-09-16T11:37:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-09-16T11:37:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:37:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool create pool_to_remove 16 16 2024-09-16T11:37:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' already exists 2024-09-16T11:37:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:37:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 pool init pool_to_remove 2024-09-16T11:37:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 mirror pool enable pool_to_remove pool 2024-09-16T11:37:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-09-16T11:37:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:37:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool create pool_to_remove 16 16 2024-09-16T11:37:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' already exists 2024-09-16T11:37:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:37:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 pool init pool_to_remove 2024-09-16T11:37:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 mirror pool enable pool_to_remove pool 2024-09-16T11:37:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster1 pool_to_remove cluster2 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster2 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster2 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-09-16T11:37:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-09-16T11:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool peer add pool_to_remove cluster2 2024-09-16T11:37:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=1be050f3-0061-4b84-94eb-9d00fb28d3be 2024-09-16T11:37:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:37:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:37:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:37:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_add cluster2 pool_to_remove cluster1 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local client_cluster=cluster1 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local remote_cluster=cluster1 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local uuid_var_name 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code 2024-09-16T11:37:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local peer_uuid 2024-09-16T11:37:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:37:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:37:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 mirror pool peer add pool_to_remove cluster1 2024-09-16T11:37:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ peer_uuid=662fd4b9-28a5-41ea-bfad-a842deea8daf 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 -eq 0 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rdp_image=test_remove_data_pool 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 pool_to_remove test 128 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling pool_to_remove/test 2024-09-16T11:37:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror test_remove_data_pool 128 --data-pool pool_to_remove 2024-09-16T11:37:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool pool_to_remove mirror/test_remove_data_pool 2024-09-16T11:37:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 pool_to_remove test 100 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:37:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench pool_to_remove/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:37:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:37:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:37:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3125.05 bytes/sec: 12 MiB/s 2024-09-16T11:37:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror test_remove_data_pool 100 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:37:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/test_remove_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:37:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:37:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:37:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 3125.05 bytes/sec: 12 MiB/s 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 pool_to_remove test 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 pool_to_remove test 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:37:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:37:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:37:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:37:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 pool_to_remove test 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush pool_to_remove/test' 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:37:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:37:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:37:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:37:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:37:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:37:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:37:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:37:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:37:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 pool_to_remove test 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '' 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:37:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:37:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-09-16T11:37:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=57110052-1f82-4c15-8623-2624bad3effc, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:37:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 pool_to_remove test 2024-09-16T11:37:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 pool_to_remove test '..*' 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=pool_to_remove 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image pool_to_remove/test 2024-09-16T11:37:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:37:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:37:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:37:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=57110052-1f82-4c15-8623-2624bad3effc, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-pool_to_remove-test.status 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:37:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position '' 2024-09-16T11:37:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:37:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status pool_to_remove/test 2024-09-16T11:37:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:37:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 0e0f4154-4677-439b-a38f-db152794c1f1 2024-09-16T11:37:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:37:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:37:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:37:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:37:16 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:37:16' 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:37:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:37:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:37:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:37:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:37:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:37:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:37:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:37:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:37:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:37:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:37:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:37:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:37:16' 2024-09-16T11:37:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 pool_to_remove test up+replaying '2024-09-16 11:37:16' 2024-09-16T11:37:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=pool_to_remove 2024-09-16T11:37:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:37:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:37:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:37:16' 2024-09-16T11:37:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:37:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:37:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status pool_to_remove --verbose --format xml 2024-09-16T11:37:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK142885660mirror.0smithi20417.2.7-1643-gd75f2382trueOK42945661mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest0e0f4154-4677-439b-a38f-db152794c1f1up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42885660mirror.0smithi2042024-09-16 11:37:16cluster25de54353-5349-4a6e-b8ce-0f29d23ba760up+stoppedlocal image is primary2024-09-16 11:37:16' 2024-09-16T11:37:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:37:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:37:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:37:16' 2024-09-16T11:37:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:37:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:37:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:37:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:37:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:37:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_remove_data_pool 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_remove_data_pool 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:37:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:37:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:37:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror test_remove_data_pool 2024-09-16T11:37:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_remove_data_pool ']' 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/test_remove_data_pool' 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_remove_data_pool 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:37:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_remove_data_pool 2024-09-16T11:37:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:37:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:37:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_remove_data_pool 2024-09-16T11:37:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test_remove_data_pool'] 2024-09-16T11:37:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:37:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_remove_data_pool 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:37:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:37:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:37:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:37:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:37:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:37:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_remove_data_pool 2024-09-16T11:37:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror test_remove_data_pool 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '' 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2024-09-16T11:37:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:37:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:37:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:37:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror test_remove_data_pool 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror test_remove_data_pool '..*' 2024-09-16T11:37:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:37:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2024-09-16T11:37:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:37:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:37:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:37:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:37:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-test_remove_data_pool.status 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:37:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:37:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:37:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:37:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position '' 2024-09-16T11:37:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:37:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:37:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:37:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:37:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:37:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test_remove_data_pool 2024-09-16T11:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_remove_data_pool: 2024-09-16T11:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 30b19db3-1bd3-418b-be76-0ce5c0f0f0e6 2024-09-16T11:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:37:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:37:14 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:37:20' 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:37:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:37:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:37:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:37:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:37:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:37:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:37:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:37:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:37:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:37:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:37:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:37:14' 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_remove_data_pool up+replaying '2024-09-16 11:37:14' 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:37:14' 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:37:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK442884295mirror.0smithi20417.2.7-1643-gd75f2382trueOK42944297mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest9b27b81f-bd43-4dc5-bbb4-0c3437048cc7up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":118}}42884295mirror.0smithi2042024-09-16 11:37:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:37:20test27f673670-80a6-4986-89e8-661d7e93cb9eup+replayingreplaying, {"bytes_per_second":35.0,"entries_behind_primary":0,"entries_per_second":0.4,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42944297mirror.1smithi2042024-09-16 11:37:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:37:20test49ccd4b16-1909-4bc0-84ae-628353f95e57up+replayingreplaying, {"bytes_per_second":153.0,"entries_behind_primary":0,"entries_per_second":1.5,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42884295mirror.0smithi2042024-09-16 11:37:10cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:37:20test_remove_data_pool30b19db3-1bd3-418b-be76-0ce5c0f0f0e6up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}42944297mirror.1smithi2042024-09-16 11:37:14cluster2575d6853-c710-4ab7-a806-f3cf450240e2up+stoppedlocal image is primary2024-09-16 11:37:20' 2024-09-16T11:37:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:37:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/last_update' 2024-09-16T11:37:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:37:14' 2024-09-16T11:37:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_remove_data_pool'\'']/state' 2024-09-16T11:37:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:37:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:37:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:37:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:37:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-09-16T11:37:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:37:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2024-09-16T11:37:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' does not exist 2024-09-16T11:37:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-09-16T11:37:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:37:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2024-09-16T11:37:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'pool_to_remove' does not exist 2024-09-16T11:37:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test_remove_data_pool 2024-09-16T11:37:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:37:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test_remove_data_pool 2024-09-16T11:37:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_remove_data_pool 2024-09-16T11:37:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test_remove_data_pool 2024-09-16T11:37:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 100% complete...done. 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test_remove_data_pool deleted 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:37:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_remove_data_pool 2024-09-16T11:37:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-09-16T11:37:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2024-09-16T11:37:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:37:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2024-09-16T11:37:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:37:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:37:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test_remove_data_pool deleted '' 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_remove_data_pool 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test_remove_data_pool 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:37:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_remove_data_pool 2024-09-16T11:37:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2024-09-16T11:37:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:37:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2024-09-16T11:37:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:37:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:37:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:37:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2024-09-16T11:37:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2024-09-16T11:37:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:37:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:37:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:37:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-09-16T11:37:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "pool_to_remove/test", 2024-09-16T11:37:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Stopped" 2024-09-16T11:37:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-09-16T11:37:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2024-09-16T11:37:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:24.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:{ 2024-09-16T11:37:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "name": "pool_to_remove/test", 2024-09-16T11:37:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Stopped" 2024-09-16T11:37:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:} 2024-09-16T11:37:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:37:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in 0 1 2 4 8 8 8 8 16 16 2024-09-16T11:37:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:37:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2024-09-16T11:37:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:37:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:37:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:37:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:37:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:37:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:37:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:37:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:37:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:34.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:37:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:37:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:37:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:37:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:37:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:37:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:37:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:37:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:37:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:37:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:37:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:37:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:37:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:37:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:37:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:37:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:37:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:37:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:37:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:37:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:37:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:37:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:38:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:38:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:38:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:38:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:38:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:38:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:38:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:38:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:38:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:38:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:38:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:38:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:38:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:38:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:38:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:38:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:38:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:38:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:38:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:38:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:38:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:38:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:38:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:38:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:38:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:38:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:38:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:38:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:38:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:38:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:38:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:38:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:38:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:38:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:38:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:38:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:38:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:38:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:38:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:38:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:38:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:38:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:38:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:38:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:38:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:38:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-09-16T11:38:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:38:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:38:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:38:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:38:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:38:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:38:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:38:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:38:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:38:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:38:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:38:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:38:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:38:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:38:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-09-16T11:39:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:12.011 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster2 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster2 2024-09-16T11:39:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:39:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:39:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:12.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:39:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:13.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:15.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:39:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:39:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:39:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:36.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:39:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:39:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:39:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:39:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:39:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:39:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:39:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:39:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:39:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:39:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:39:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:39:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:39:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:39:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:39:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:39:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:39:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:39:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:39:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:40:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:40:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:40:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:40:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:40:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:40:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:40:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:40:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:40:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:40:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:40:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:40:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:40:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:40:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:40:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:40:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:40:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:40:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:40:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:40:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:40:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:40:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:40:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:40:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:40:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:40:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:40:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:40:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:40:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:40:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:40:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-09-16T11:40:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:40:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:40:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:40:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:40:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:40:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:40:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:40:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:40:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:40:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:40:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:41.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:40:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 16 2024-09-16T11:40:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster2 '!=' cluster2 ']' 2024-09-16T11:40:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:40:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2024-09-16T11:40:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:40:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:40:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:40:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:40:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:40:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2024-09-16T11:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:40:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:40:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:40:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:40:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2024-09-16T11:40:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2024-09-16T11:40:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:40:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:40:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:40:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: snapshot rename' 2024-09-16T11:40:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:40:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:40:57 TEST: snapshot rename 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:40:57 TEST: snapshot rename 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ snap_name=snap_rename 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_snapshot cluster2 mirror test2 snap_rename_0 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_0 2024-09-16T11:40:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap_rename_0 2024-09-16T11:40:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-16T11:40:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 1 20 2024-09-16T11:40:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 1 - 1 2024-09-16T11:40:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_0 snap_rename_1 2024-09-16T11:40:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_0 2024-09-16T11:40:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_1 2024-09-16T11:40:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_0 mirror/test2@snap_rename_1 2024-09-16T11:40:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 2 - 1 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_1 snap_rename_2 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_1 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_2 2024-09-16T11:40:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_1 mirror/test2@snap_rename_2 2024-09-16T11:40:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 3 - 1 2024-09-16T11:40:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_2 snap_rename_3 2024-09-16T11:40:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_2 2024-09-16T11:40:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_3 2024-09-16T11:40:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_2 mirror/test2@snap_rename_3 2024-09-16T11:40:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 4 - 1 2024-09-16T11:40:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_3 snap_rename_4 2024-09-16T11:40:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_3 2024-09-16T11:40:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_4 2024-09-16T11:40:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_3 mirror/test2@snap_rename_4 2024-09-16T11:40:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 5 - 1 2024-09-16T11:40:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_4 snap_rename_5 2024-09-16T11:40:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_4 2024-09-16T11:40:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_5 2024-09-16T11:40:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_4 mirror/test2@snap_rename_5 2024-09-16T11:40:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 6 - 1 2024-09-16T11:40:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_5 snap_rename_6 2024-09-16T11:40:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_5 2024-09-16T11:40:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_6 2024-09-16T11:40:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_5 mirror/test2@snap_rename_6 2024-09-16T11:40:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 7 - 1 2024-09-16T11:40:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_6 snap_rename_7 2024-09-16T11:40:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_6 2024-09-16T11:40:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_7 2024-09-16T11:40:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_6 mirror/test2@snap_rename_7 2024-09-16T11:40:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 8 - 1 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_7 snap_rename_8 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_7 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_8 2024-09-16T11:40:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_7 mirror/test2@snap_rename_8 2024-09-16T11:40:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 9 - 1 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_8 snap_rename_9 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_8 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_9 2024-09-16T11:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_8 mirror/test2@snap_rename_9 2024-09-16T11:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 10 - 1 2024-09-16T11:40:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_9 snap_rename_10 2024-09-16T11:40:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_9 2024-09-16T11:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_10 2024-09-16T11:40:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_9 mirror/test2@snap_rename_10 2024-09-16T11:40:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 11 - 1 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_10 snap_rename_11 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_10 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_11 2024-09-16T11:40:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_10 mirror/test2@snap_rename_11 2024-09-16T11:40:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 12 - 1 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_11 snap_rename_12 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_11 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_12 2024-09-16T11:40:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_11 mirror/test2@snap_rename_12 2024-09-16T11:40:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 13 - 1 2024-09-16T11:40:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_12 snap_rename_13 2024-09-16T11:40:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_12 2024-09-16T11:40:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_13 2024-09-16T11:40:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_12 mirror/test2@snap_rename_13 2024-09-16T11:40:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 14 - 1 2024-09-16T11:40:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_13 snap_rename_14 2024-09-16T11:40:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_13 2024-09-16T11:40:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_14 2024-09-16T11:40:58.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_13 mirror/test2@snap_rename_14 2024-09-16T11:40:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 15 - 1 2024-09-16T11:40:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_14 snap_rename_15 2024-09-16T11:40:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_14 2024-09-16T11:40:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_15 2024-09-16T11:40:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_14 mirror/test2@snap_rename_15 2024-09-16T11:40:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 16 - 1 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_15 snap_rename_16 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_15 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_16 2024-09-16T11:40:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_15 mirror/test2@snap_rename_16 2024-09-16T11:40:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 17 - 1 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_16 snap_rename_17 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_16 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_17 2024-09-16T11:40:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_16 mirror/test2@snap_rename_17 2024-09-16T11:40:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 18 - 1 2024-09-16T11:40:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_17 snap_rename_18 2024-09-16T11:40:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_17 2024-09-16T11:40:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_18 2024-09-16T11:40:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_17 mirror/test2@snap_rename_18 2024-09-16T11:40:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 19 - 1 2024-09-16T11:40:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_18 snap_rename_19 2024-09-16T11:40:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_18 2024-09-16T11:40:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_19 2024-09-16T11:40:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_18 mirror/test2@snap_rename_19 2024-09-16T11:40:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in `seq 1 20` 2024-09-16T11:40:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ expr 20 - 1 2024-09-16T11:40:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_19 snap_rename_20 2024-09-16T11:40:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:40:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap_rename_19 2024-09-16T11:40:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local new_snap=snap_rename_20 2024-09-16T11:40:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_19 mirror/test2@snap_rename_20 2024-09-16T11:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_snap_present cluster1 mirror test2 snap_rename_20 2024-09-16T11:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap_name=snap_rename_20 2024-09-16T11:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-09-16T11:41:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:41:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:41:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:41:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-09-16T11:41:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:41:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:41:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:41:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-09-16T11:41:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:41:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:41:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:41:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-09-16T11:41:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:error setting snapshot context: (2) No such file or directory 2024-09-16T11:41:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ continue 2024-09-16T11:41:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T11:41:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:41:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test2': 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 3 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 10c9a3290aa0 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.10c9a3290aa0 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:41:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:36:48 2024 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:36:48 2024 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:36:48 2024 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: protected: False 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 10c9a3290aa0 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: 7f673670-80a6-4986-89e8-661d7e93cb9e 2024-09-16T11:41:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: false 2024-09-16T11:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test4 snap1 2024-09-16T11:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap1 2024-09-16T11:41:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap1 2024-09-16T11:41:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ unprotect_snapshot cluster2 mirror test4 snap2 2024-09-16T11:41:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:41:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local snap=snap2 2024-09-16T11:41:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test4@snap2 2024-09-16T11:41:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-09-16T11:41:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test2 2024-09-16T11:41:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test2 2024-09-16T11:41:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test2 2024-09-16T11:41:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test2 2024-09-16T11:41:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing all snapshots: 33% complete... Removing all snapshots: 66% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-16T11:41:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test2 2024-09-16T11:41:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:41:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for i in ${image2} ${image4} 2024-09-16T11:41:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test4 2024-09-16T11:41:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:41:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:41:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:41:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test4 2024-09-16T11:41:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test4 2024-09-16T11:41:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test4 2024-09-16T11:41:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-16T11:41:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test4 2024-09-16T11:41:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:41:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: disable mirror while daemon is stopped' 2024-09-16T11:41:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:41:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:41:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:41:26 TEST: disable mirror while daemon is stopped 2024-09-16T11:41:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:41:26 TEST: disable mirror while daemon is stopped 2024-09-16T11:41:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-09-16T11:41:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:41:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:41:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:41:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:41:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=32525 2024-09-16T11:41:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 32525 ']' 2024-09-16T11:41:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 32525 2024-09-16T11:41:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:41:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=32525 '$2 == pid {print; exit 1}' 2024-09-16T11:41:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:ubuntu 32525 0.5 0.3 622276 100696 ? Ssl 11:24 0:05 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:41:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:41:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=32525 '$2 == pid {print; exit 1}' 2024-09-16T11:41:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:41:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=32525 '$2 == pid {print; exit 1}' 2024-09-16T11:41:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:41:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:41:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:41:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-09-16T11:41:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:41:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:41:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:41:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=32553 2024-09-16T11:41:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 32553 ']' 2024-09-16T11:41:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 32553 2024-09-16T11:41:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:41:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=32553 '$2 == pid {print; exit 1}' 2024-09-16T11:41:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:41:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=32553 '$2 == pid {print; exit 1}' 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:41:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:41:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-16T11:41:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:41:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-09-16T11:41:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:41:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:41:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:41:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster2 2024-09-16T11:41:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:41:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:41:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:0 '' 2024-09-16T11:41:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:41:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:41:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:41:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:0 2024-09-16T11:41:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:0 2024-09-16T11:41:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:41:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:41:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-16T11:41:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:41:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:41:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=36070 2024-09-16T11:41:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 36070 ']' 2024-09-16T11:41:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 36070 2024-09-16T11:41:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:41:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=36070 '$2 == pid {print; exit 1}' 2024-09-16T11:41:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:41:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=36070 '$2 == pid {print; exit 1}' 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:41:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:41:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:41:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:41:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:41:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:41:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:41:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:41:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:0 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-16T11:41:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:41:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:1 '' 2024-09-16T11:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:1 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:1 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:41:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:41:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-16T11:41:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:41:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:41:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=36103 2024-09-16T11:41:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 36103 ']' 2024-09-16T11:41:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 36103 2024-09-16T11:41:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:41:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=36103 '$2 == pid {print; exit 1}' 2024-09-16T11:41:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:41:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:41:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=36103 '$2 == pid {print; exit 1}' 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:41:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:41:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:41:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:41:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:41:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:41:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:41:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:1 2024-09-16T11:41:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-16T11:41:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:41:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:41:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-16T11:41:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=image 2024-09-16T11:41:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-16T11:41:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from pool to image 2024-09-16T11:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror test 2024-09-16T11:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test 2024-09-16T11:41:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c75f0ead86 2024-09-16T11:41:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c75f0ead86 2024-09-16T11:41:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 10c75f0ead86 2024-09-16T11:41:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:41:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:41:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-09-16T11:41:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:41:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:41:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:41:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:41:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=10c75f0ead86 2024-09-16T11:41:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:41:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c75f0ead86 2024-09-16T11:41:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c75f0ead86 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c75f0ead86 2024-09-16T11:41:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c75f0ead86 2024-09-16T11:41:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c75f0ead86 -o 10c75f0ead86 = 10c75f0ead86 2024-09-16T11:41:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:41:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:41:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c75f0ead86 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c75f0ead86 2024-09-16T11:41:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c75f0ead86 2024-09-16T11:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c75f0ead86 2024-09-16T11:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c75f0ead86 -o 10c75f0ead86 = 10c75f0ead86 2024-09-16T11:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:41:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c75f0ead86 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c75f0ead86 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=10c75f0ead86 2024-09-16T11:41:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 10c75f0ead86 2024-09-16T11:41:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 10c75f0ead86 -o 10c75f0ead86 = 10c75f0ead86 2024-09-16T11:41:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:41:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:41:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:41:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 10c75f0ead86 2024-09-16T11:41:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=10c75f0ead86 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=pool 2024-09-16T11:41:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2024-09-16T11:41:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:note: changing mirroring mode from image to pool 2024-09-16T11:41:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_journaling cluster2 mirror test 2024-09-16T11:41:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 feature enable mirror/test journaling 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:41:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:41:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:41:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:41:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:41:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:41:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:41:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:41:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:41:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:41:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:41:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:41:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:41:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:41:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:41:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:41:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=165153170058 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 165153170058 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 165153170058 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:41:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:41:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:41:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:41:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:41:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:41:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:41:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:41:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:41:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:41:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:41:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:41:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:41:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:41:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:41:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:41:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:41:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:41:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:41:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:41:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: non-default namespace image mirroring' 2024-09-16T11:41:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:41:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:41:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:41:50 TEST: non-default namespace image mirroring 2024-09-16T11:41:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:41:50 TEST: non-default namespace image mirroring 2024-09-16T11:41:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay' 2024-09-16T11:41:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:41:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:41:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:41:50 - replay 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:41:50 - replay 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror/ns1 test 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:41:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test 2024-09-16T11:41:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror/ns2 test 2024-09-16T11:41:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:41:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns2/test 2024-09-16T11:41:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ enable_mirror cluster2 mirror/ns2 test journal 2024-09-16T11:41:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local mode=journal 2024-09-16T11:41:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns2/test journal 2024-09-16T11:41:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring enabled 2024-09-16T11:41:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 info mirror/ns2/test 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:rbd image 'test': 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: size 128 MiB in 32 objects 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: order 22 (4 MiB objects) 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: snapshot_count: 0 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: id: 17facc33b002 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: block_name_prefix: rbd_data.17facc33b002 2024-09-16T11:41:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: format: 2 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: features: layering, exclusive-lock, journaling 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: op_features: 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: flags: 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: create_timestamp: Mon Sep 16 11:41:50 2024 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: access_timestamp: Mon Sep 16 11:41:50 2024 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: modify_timestamp: Mon Sep 16 11:41:50 2024 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: journal: 17facc33b002 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring state: enabled 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring mode: journal 2024-09-16T11:41:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring global id: d231a2b3-7e34-45ca-b82b-c006b2a3b8ae 2024-09-16T11:41:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: mirroring primary: true 2024-09-16T11:41:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test 2024-09-16T11:41:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started 2024-09-16T11:41:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:41:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:41:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started 2024-09-16T11:41:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:41:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:41:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2024-09-16T11:41:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-09-16T11:41:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:41:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2024-09-16T11:41:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-09-16T11:41:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:41:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:41:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:41:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2024-09-16T11:41:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-09-16T11:41:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:41:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2024-09-16T11:41:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:41:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:41:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:41:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:41:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:41:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2024-09-16T11:41:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2024-09-16T11:41:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:41:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:41:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:41:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2024-09-16T11:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2024-09-16T11:41:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:41:55.182 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:41:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror/ns2 test 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror/ns2 test started 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:41:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror/ns2 test started 2024-09-16T11:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns2/test 2024-09-16T11:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:41:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:41:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:41:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns2/test 2024-09-16T11:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:41:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:41:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:41:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:41:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:41:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:41:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns2/test 2024-09-16T11:41:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:41:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:41:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:41:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:41:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror/ns1 test 100 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:41:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:41:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:41:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:41:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:41:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:41:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2777.82 bytes/sec: 11 MiB/s 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror/ns2 test 100 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:41:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/ns2/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:41:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:41:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:41:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2857.19 bytes/sec: 11 MiB/s 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:41:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:41:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:41:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:41:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test' 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:41:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:41:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:41:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:41:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2024-09-16T11:41:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '' 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:41:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-09-16T11:41:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-09-16T11:41:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.status 2024-09-16T11:41:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-09-16T11:41:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.status 2024-09-16T11:41:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:41:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:41:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:41:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.status 2024-09-16T11:41:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test '..*' 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:41:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test.status 2024-09-16T11:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test.status 2024-09-16T11:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.status 2024-09-16T11:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test 2024-09-16T11:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.status 2024-09-16T11:41:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:41:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:41:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:41:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.status 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns2 test 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns2 test 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:41:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:41:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns2 test 2024-09-16T11:41:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test ']' 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns2/test' 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns2/test 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:41:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:41:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:41:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns2/test 2024-09-16T11:41:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:41:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:41:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:41:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:41:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:41:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:41:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns2/test 2024-09-16T11:41:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns2 test 2024-09-16T11:41:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns2 test '' 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns2-test.status 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns2-test.status 2024-09-16T11:41:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.status 2024-09-16T11:41:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test 2024-09-16T11:41:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.status 2024-09-16T11:41:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:41:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:41:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:41:57.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.status 2024-09-16T11:41:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:41:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns2 test 2024-09-16T11:41:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns2 test '..*' 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns2-test.status 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns2-test.status 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.status 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns2/test 2024-09-16T11:41:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.status 2024-09-16T11:41:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:41:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:41:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:41:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:41:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.status 2024-09-16T11:41:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position '' 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:41:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:41:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test 2024-09-16T11:41:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 30c87c99-0c9f-4395-8db2-f3920d326e86 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:41:53 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:41:58.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:41:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:41:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:41:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:41:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:41:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:41:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:41:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:41:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:41:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:41:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:41:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:41:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:41:53' 2024-09-16T11:41:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying '2024-09-16 11:41:53' 2024-09-16T11:41:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:41:53' 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:41:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2024-09-16T11:41:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK157065713mirror.0smithi20417.2.7-1643-gd75f2382trueOK57125719mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest30c87c99-0c9f-4395-8db2-f3920d326e86up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57065713mirror.0smithi2042024-09-16 11:41:53cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:41:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:41:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:41:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:41:53' 2024-09-16T11:41:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:41:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:41:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:41:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:41:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:41:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:41:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:41:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:41:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:41:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position '' 2024-09-16T11:41:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:41:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns2/test 2024-09-16T11:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: d231a2b3-7e34-45ca-b82b-c006b2a3b8ae 2024-09-16T11:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:41:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:41:54 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:41:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:41:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:41:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:41:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:41:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:41:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:41:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:41:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:41:54' 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns2 test up+replaying '2024-09-16 11:41:54' 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:41:54' 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:41:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns2 --verbose --format xml 2024-09-16T11:41:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK157065713mirror.0smithi20417.2.7-1643-gd75f2382trueOK57125719mirror.1smithi20417.2.7-1643-gd75f2382falseOKtestd231a2b3-7e34-45ca-b82b-c006b2a3b8aeup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57065713mirror.0smithi2042024-09-16 11:41:54cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:41:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:41:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:41:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:41:54' 2024-09-16T11:41:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:41:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:41:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:41:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:41:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror/ns1 test 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:41:59.504 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror/ns1-test.export 2024-09-16T11:41:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror/ns1-test.export 2024-09-16T11:41:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.export 2024-09-16T11:41:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror/ns1-test.export 2024-09-16T11:41:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror/ns1-test.export 2024-09-16T11:41:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test.export 2024-09-16T11:41:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test.export 2024-09-16T11:41:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.export 2024-09-16T11:41:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test.export 2024-09-16T11:41:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:41:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test.export 2024-09-16T11:41:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test.export 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror/ns2 test 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror/ns2-test.export 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror/ns2-test.export 2024-09-16T11:41:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.export 2024-09-16T11:41:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror/ns2-test.export 2024-09-16T11:41:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror/ns2-test.export 2024-09-16T11:41:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:41:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror_ns2-test.export 2024-09-16T11:41:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns2-test.export 2024-09-16T11:41:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/ns2/test /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.export 2024-09-16T11:42:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/ns2/test /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns2-test.export 2024-09-16T11:42:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns2-test.export 2024-09-16T11:42:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns2-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns2-test.export 2024-09-16T11:42:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - disable mirroring / delete image' 2024-09-16T11:42:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:00 - disable mirroring / delete image 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:00 - disable mirroring / delete image 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror/ns1 test 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:42:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror/ns1 test 2024-09-16T11:42:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test 2024-09-16T11:42:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test 2024-09-16T11:42:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disable_mirror cluster2 mirror/ns2 test 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/ns2/test 2024-09-16T11:42:00.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Mirroring disabled 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns1 test 2024-09-16T11:42:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2024-09-16T11:42:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:42:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror/ns1 test deleted '' 2024-09-16T11:42:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns1 test 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns1/test 2024-09-16T11:42:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror/ns2 test deleted 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns2 test 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2024-09-16T11:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:42:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror/ns2 test deleted '' 2024-09-16T11:42:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:42:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror/ns2 test 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns2 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/ns2/test 2024-09-16T11:42:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror/ns2 test 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror/ns2 test 2024-09-16T11:42:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns2 2024-09-16T11:42:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns2/test 2024-09-16T11:42:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/ns2/test 2024-09-16T11:42:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:42:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - data pool' 2024-09-16T11:42:01.003 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:01 - data pool 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:01 - data pool 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ dp_image=test_data_pool 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror/ns1 test_data_pool 128 --data-pool mirror_parent 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-09-16T11:42:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/ns1/test_data_pool 2024-09-16T11:42:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 info mirror/ns1/test_data_pool 2024-09-16T11:42:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-09-16T11:42:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-09-16T11:42:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test_data_pool 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test_data_pool started 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror/ns1 test_data_pool started 2024-09-16T11:42:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:42:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:42:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2024-09-16T11:42:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_data_pool cluster1 mirror/ns1 test_data_pool 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 info mirror/ns1/test_data_pool 2024-09-16T11:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ awk '$1 == "data_pool:" {print $2}' 2024-09-16T11:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ data_pool=mirror_parent 2024-09-16T11:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test mirror_parent = mirror_parent 2024-09-16T11:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror/ns1 test_data_pool 100 2024-09-16T11:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=100 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2024-09-16T11:42:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2024-09-16T11:42:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:42:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 100 ops/sec: 2941.22 bytes/sec: 11 MiB/s 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:42:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:42:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:42:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:42:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2024-09-16T11:42:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool' 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:42:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '' 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2024-09-16T11:42:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:42:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:42:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:42:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*' 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2024-09-16T11:42:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:42:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2024-09-16T11:42:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:42:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:42:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' '!=' '' 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=3, tag_tid=1, entry_tid=3' = '' 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:42:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=2 2024-09-16T11:42:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:42:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:42:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:42:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=1 2024-09-16T11:42:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:42:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:42:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:42:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=101 2024-09-16T11:42:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=3, tag_tid=1, entry_tid=3' 2024-09-16T11:42:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:42:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:42:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=3 2024-09-16T11:42:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 2 = 1 -a 101 -le 3 2024-09-16T11:42:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:42:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:42:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n test_data_pool ']' 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/ns1/test_data_pool' 2024-09-16T11:42:06.474 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2024-09-16T11:42:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:42:06.559 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2024-09-16T11:42:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '' 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2024-09-16T11:42:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:42:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:42:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:42:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror/ns1 test_data_pool '..*' 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror/ns1 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test_data_pool 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror/ns1-test_data_pool.status 2024-09-16T11:42:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2024-09-16T11:42:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:42:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2024-09-16T11:42:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.status 2024-09-16T11:42:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:42:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=1, tag_tid=2, entry_tid=101' -a 'object_number=1, tag_tid=2, entry_tid=101' = 'object_number=1, tag_tid=2, entry_tid=101' 2024-09-16T11:42:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position 2024-09-16T11:42:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test_data_pool up+replaying primary_position '' 2024-09-16T11:42:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/ns1/test_data_pool 2024-09-16T11:42:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test_data_pool: 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 4dc760b0-668f-48ed-b77c-c3e8754e9e1b 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:42:03 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:42:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:42:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:42:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:42:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:42:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:42:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:42:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:42:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:42:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:42:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:42:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:03' 2024-09-16T11:42:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test_data_pool up+replaying '2024-09-16 11:42:03' 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:42:03' 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2024-09-16T11:42:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='OKOKOK157065713mirror.0smithi20417.2.7-1643-gd75f2382trueOK57125719mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest_data_pool4dc760b0-668f-48ed-b77c-c3e8754e9e1bup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}57125719mirror.1smithi2042024-09-16 11:42:03cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:42:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:42:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/last_update' 2024-09-16T11:42:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:03' 2024-09-16T11:42:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test_data_pool'\'']/state' 2024-09-16T11:42:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:42:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:42:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror/ns1 test_data_pool 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror/ns1-test_data_pool.export 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror/ns1-test_data_pool.export 2024-09-16T11:42:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.export 2024-09-16T11:42:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror/ns1-test_data_pool.export 2024-09-16T11:42:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror/ns1-test_data_pool.export 2024-09-16T11:42:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test_data_pool.export 2024-09-16T11:42:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test_data_pool.export 2024-09-16T11:42:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/ns1/test_data_pool /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.export 2024-09-16T11:42:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/ns1/test_data_pool /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test_data_pool.export 2024-09-16T11:42:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test_data_pool.export 2024-09-16T11:42:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror_ns1-test_data_pool.export /tmp/tmp.MThcdnqr7J/cluster1-mirror_ns1-test_data_pool.export 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:42:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror/ns1 test_data_pool 2024-09-16T11:42:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror/ns1 2024-09-16T11:42:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test_data_pool 2024-09-16T11:42:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test_data_pool 2024-09-16T11:42:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test_data_pool 2024-09-16T11:42:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:42:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: simple image resync' 2024-09-16T11:42:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:08 TEST: simple image resync 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:08 TEST: simple image resync 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror test image_id 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''165153170058'\''' 2024-09-16T11:42:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=165153170058 2024-09-16T11:42:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-09-16T11:42:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 165153170058 2024-09-16T11:42:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2024-09-16T11:42:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-09-16T11:42:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 165153170058 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=165153170058 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 165153170058 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 165153170058 2024-09-16T11:42:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=165153170058 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:08.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:08.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:42:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:42:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:42:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:42:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:42:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present '' 2024-09-16T11:42:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=1651bcb67e44 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1651bcb67e44 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 1651bcb67e44 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:42:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:42:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:42:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:42:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52b6ae03-449a-475f-831c-c7d3f2f3e072 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:42:14 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:42:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:42:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:42:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:42:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:42:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:42:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:42:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:42:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:42:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:42:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:42:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:42:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:42:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:42:18.306 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:42:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:14' 2024-09-16T11:42:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:42:14' 2024-09-16T11:42:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:42:14' 2024-09-16T11:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:42:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING157065713mirror.0smithi20417.2.7-1643-gd75f2382trueOK57125719mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest52b6ae03-449a-475f-831c-c7d3f2f3e072up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}57065713mirror.0smithi2042024-09-16 11:42:14cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:42:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:42:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:42:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:14' 2024-09-16T11:42:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:42:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:42:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:42:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:42:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:42:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:42:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:42:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:42:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:18.801 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: image resync while replayer is stopped' 2024-09-16T11:42:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:18 TEST: image resync while replayer is stopped 2024-09-16T11:42:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:18 TEST: image resync while replayer is stopped 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/test 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/test 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/test 2024-09-16T11:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2024-09-16T11:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test stopped 2024-09-16T11:42:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:42:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:42:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:42:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:42:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:42:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror test image_id 2024-09-16T11:42:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''1651bcb67e44'\''' 2024-09-16T11:42:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=1651bcb67e44 2024-09-16T11:42:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-09-16T11:42:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 1651bcb67e44 2024-09-16T11:42:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2024-09-16T11:42:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-09-16T11:42:20.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 1651bcb67e44 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:42:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=1651bcb67e44 2024-09-16T11:42:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 1651bcb67e44 2024-09-16T11:42:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 1651bcb67e44 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=1651bcb67e44 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2024-09-16T11:42:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=16514f207a31 2024-09-16T11:42:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present 16514f207a31 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16514f207a31 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16514f207a31 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16514f207a31 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16514f207a31 -o 16514f207a31 = 16514f207a31 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:42:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:42:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:42:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:42:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T11:42:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T11:42:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:42:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:42:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:42:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52b6ae03-449a-475f-831c-c7d3f2f3e072 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:42:21 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:42:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:42:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:42:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:42:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:42:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:42:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:42:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:42:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2024-09-16T11:42:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:42:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:42:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:42:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:42:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:42:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:42:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:21' 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:42:21' 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:42:21' 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:42:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING157065713mirror.0smithi20417.2.7-1643-gd75f2382trueOK57125719mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest52b6ae03-449a-475f-831c-c7d3f2f3e072up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}}57065713mirror.0smithi2042024-09-16 11:42:21cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:42:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:42:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:42:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:21' 2024-09-16T11:42:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:42:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:42:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:42:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:42:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:42:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:42:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:42:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:42:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: request image resync while daemon is offline' 2024-09-16T11:42:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:24 TEST: request image resync while daemon is offline 2024-09-16T11:42:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:24 TEST: request image resync while daemon is offline 2024-09-16T11:42:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-09-16T11:42:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:42:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T11:42:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:42:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:42:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:42:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:42:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:42:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=66415 2024-09-16T11:42:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 66415 ']' 2024-09-16T11:42:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 66415 2024-09-16T11:42:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:42:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:42:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=66415 '$2 == pid {print; exit 1}' 2024-09-16T11:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:42:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=66415 '$2 == pid {print; exit 1}' 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-16T11:42:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:42:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:42:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:42:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T11:42:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:42:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:42:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:42:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:42:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:42:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=66443 2024-09-16T11:42:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 66443 ']' 2024-09-16T11:42:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 66443 2024-09-16T11:42:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:42:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:42:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=66443 '$2 == pid {print; exit 1}' 2024-09-16T11:42:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:42:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:42:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=66443 '$2 == pid {print; exit 1}' 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-09-16T11:42:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:42:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:42:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror test image_id 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16514f207a31'\''' 2024-09-16T11:42:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16514f207a31 2024-09-16T11:42:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-09-16T11:42:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16514f207a31 2024-09-16T11:42:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/test 2024-09-16T11:42:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-09-16T11:42:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster1 2024-09-16T11:42:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:0 2024-09-16T11:42:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster1:1 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test deleted 16514f207a31 2024-09-16T11:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16514f207a31 2024-09-16T11:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16514f207a31 2024-09-16T11:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16514f207a31 2024-09-16T11:42:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16514f207a31 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16514f207a31 2024-09-16T11:42:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16514f207a31 2024-09-16T11:42:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16514f207a31 -o 16514f207a31 = 16514f207a31 2024-09-16T11:42:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:42:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16514f207a31 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16514f207a31 2024-09-16T11:42:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16514f207a31 2024-09-16T11:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16514f207a31 2024-09-16T11:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16514f207a31 -o 16514f207a31 = 16514f207a31 2024-09-16T11:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:42:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16514f207a31 2024-09-16T11:42:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16514f207a31 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:30.043 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16514f207a31 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16514f207a31 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16514f207a31 -o 16514f207a31 = 16514f207a31 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test deleted 16514f207a31 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16514f207a31 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16eb5fddd7cd 2024-09-16T11:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16eb5fddd7cd 2024-09-16T11:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16514f207a31 -o 16514f207a31 = 16eb5fddd7cd 2024-09-16T11:42:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror test present 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=16eb5fddd7cd 2024-09-16T11:42:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror test present 16eb5fddd7cd 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16eb5fddd7cd 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror test 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=test 2024-09-16T11:42:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T11:42:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16eb5fddd7cd 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16eb5fddd7cd 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16eb5fddd7cd -o 16eb5fddd7cd = 16eb5fddd7cd 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T11:42:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T11:42:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:35.567 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T11:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position '' 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T11:42:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='test: 2024-09-16T11:42:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 52b6ae03-449a-475f-831c-c7d3f2f3e072 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:42:33 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:42:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:42:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:42:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:42:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2024-09-16T11:42:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:42:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:42:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:42:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:42:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:42:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:42:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:33' 2024-09-16T11:42:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 11:42:33' 2024-09-16T11:42:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:42:33' 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:42:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKtest52b6ae03-449a-475f-831c-c7d3f2f3e072up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}}58615867mirror.0smithi2042024-09-16 11:42:33cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:42:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:42:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T11:42:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:33' 2024-09-16T11:42:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T11:42:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:42:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:42:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:42:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror test 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-test.export 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-test.export 2024-09-16T11:42:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:42:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-test.export 2024-09-16T11:42:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-test.export 2024-09-16T11:42:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export 2024-09-16T11:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:37.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-test.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-test.export 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror test 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:42:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror test 2024-09-16T11:42:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=test 2024-09-16T11:42:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-16T11:42:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-16T11:42:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: client disconnect' 2024-09-16T11:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:37 TEST: client disconnect 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:37 TEST: client disconnect 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=laggy 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror laggy 128 --journal-object-size 64K 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 128 ']' 2024-09-16T11:42:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=128 2024-09-16T11:42:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --journal-object-size 64K mirror/laggy 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror laggy 10 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=10 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:42:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2024-09-16T11:42:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2024-09-16T11:42:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:42:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 10 ops/sec: 1428.59 bytes/sec: 5.6 MiB/s 2024-09-16T11:42:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay stopped after disconnect' 2024-09-16T11:42:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:37 - replay stopped after disconnect 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:37 - replay stopped after disconnect 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-09-16T11:42:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:42:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:42:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:42:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:42:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:42:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:42:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:42:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 2 2024-09-16T11:42:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:42:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:41.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:42:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:42:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:42:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:42:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:42:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:42:42.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:42:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:42:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' -a 'object_number=3, tag_tid=2, entry_tid=3' = 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=2, entry_tid=3' 2024-09-16T11:42:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disconnect_image cluster2 mirror laggy 2024-09-16T11:42:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2024-09-16T11:42:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:client 88cac606-ed81-47bf-b993-eb31e1d47ba4 disconnected 2024-09-16T11:42:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-16T11:42:42.662 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=disconnected] 2024-09-16T11:42:42.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:42:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:42:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:42:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:42:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:42:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:42:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:42:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:42:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:42:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2024-09-16T11:42:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='laggy: 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a6f1aea9-096b-4f04-b89b-59c015cd8d84 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: disconnected 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:42:42 2024-09-16T11:42:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:42:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:42:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:42:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:42:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:42:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-09-16T11:42:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-09-16T11:42:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*disconnected' 2024-09-16T11:42:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: disconnected 2024-09-16T11:42:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:42:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:42:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:42:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:42:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:42:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:42:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:42:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:42:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:42' 2024-09-16T11:42:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-09-16 11:42:42' 2024-09-16T11:42:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:42:42' 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:42:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:42:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKlaggya6f1aea9-096b-4f04-b89b-59c015cd8d84up+errordisconnected58685871mirror.1smithi2042024-09-16 11:42:42cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:42:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:42:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2024-09-16T11:42:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:42:42' 2024-09-16T11:42:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2024-09-16T11:42:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-09-16T11:42:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:42:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-09-16T11:42:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:42:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay started after resync requested' 2024-09-16T11:42:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:44 - replay started after resync requested 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:44 - replay started after resync requested 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror laggy image_id 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16efc4b24b8d'\''' 2024-09-16T11:42:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16efc4b24b8d 2024-09-16T11:42:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-09-16T11:42:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16efc4b24b8d 2024-09-16T11:42:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy 2024-09-16T11:42:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-09-16T11:42:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 16efc4b24b8d 2024-09-16T11:42:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:42:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16efc4b24b8d 2024-09-16T11:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16efc4b24b8d 2024-09-16T11:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16efc4b24b8d 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16efc4b24b8d 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16efc4b24b8d 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16efc4b24b8d 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16efc4b24b8d -o 16efc4b24b8d = 16efc4b24b8d 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16efc4b24b8d 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16efc4b24b8d 2024-09-16T11:42:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16efc4b24b8d 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16efc4b24b8d 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16efc4b24b8d -o 16efc4b24b8d = 16efc4b24b8d 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:42:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16efc4b24b8d 2024-09-16T11:42:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16efc4b24b8d 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy present 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:42:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:42:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:42:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:42:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:42:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:42:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:42:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:42:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:42:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:42:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:42:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:42:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:42:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16ef642a70c1 2024-09-16T11:42:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef642a70c1 2024-09-16T11:42:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 16ef642a70c1 2024-09-16T11:42:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:42:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:42:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:42:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:42:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:42:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:42:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:42:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:42:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:42:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:42:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:42:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:42:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:42:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:42:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:42:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:42:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:42:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=3, tag_tid=3, entry_tid=3' 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=3, tag_tid=3, entry_tid=3' 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' -a 'object_number=3, tag_tid=3, entry_tid=3' = 'object_number=3, tag_tid=3, entry_tid=3' 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' 2024-09-16T11:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror laggy 2024-09-16T11:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:42:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-laggy.export 2024-09-16T11:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-laggy.export 2024-09-16T11:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export 2024-09-16T11:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-laggy.export 2024-09-16T11:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-laggy.export 2024-09-16T11:42:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:42:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export 2024-09-16T11:42:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:42:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:42:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:42:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:42:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - disconnected after max_concurrent_object_sets reached' 2024-09-16T11:42:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:57 - disconnected after max_concurrent_object_sets reached 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:57 - disconnected after max_concurrent_object_sets reached 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror stop mirror/laggy 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror/laggy 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:57.839 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror/laggy 2024-09-16T11:42:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror stop : stop rbd mirror 2024-09-16T11:42:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror/laggy 2024-09-16T11:42:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror/laggy 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:42:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:42:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:42:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:42:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:42:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:42:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:42:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:42:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:42:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:42:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:42:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:42:59.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:42:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:42:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:42:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:42:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:42:59.211 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:59.212 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 0 2024-09-16T11:42:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 0 2024-09-16T11:42:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2024-09-16T11:42:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=3, tag_tid=3, entry_tid=3' 2024-09-16T11:42:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 1 2024-09-16T11:42:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets 2024-09-16T11:42:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=1 2024-09-16T11:42:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 1 2024-09-16T11:42:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror laggy 20 16384 2024-09-16T11:42:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=20 2024-09-16T11:42:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=16384 2024-09-16T11:42:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16384 2024-09-16T11:42:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand 2024-09-16T11:42:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 16384 io_threads 1 bytes 327680 pattern random 2024-09-16T11:42:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:42:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 20 ops/sec: 1333.35 bytes/sec: 21 MiB/s 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster2 mirror laggy 20 16384 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=20 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=16384 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16384 2024-09-16T11:42:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 bench mirror/laggy --io-type write --io-size 16384 --io-threads 1 --io-total 327680 --io-pattern rand 2024-09-16T11:42:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 16384 io_threads 1 bytes 327680 pattern random 2024-09-16T11:42:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:42:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 20 ops/sec: 1333.35 bytes/sec: 21 MiB/s 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:42:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:42:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 1 2024-09-16T11:42:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:42:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:42:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=6, entry_tid=41], [object_number=16, tag_tid=6, entry_tid=40], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected] 2024-09-16T11:42:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=disconnected] 2024-09-16T11:42:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:42:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' 2024-09-16T11:42:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_journal_max_concurrent_object_sets 0 2024-09-16T11:42:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:42:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_journal_max_concurrent_object_sets 2024-09-16T11:42:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=0 2024-09-16T11:42:59.501 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_journal_max_concurrent_object_sets 0 2024-09-16T11:42:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay is still stopped (disconnected) after restart' 2024-09-16T11:42:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:42:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:42:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:42:59 - replay is still stopped '(disconnected)' after restart 2024-09-16T11:42:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:42:59 - replay is still stopped (disconnected) after restart 2024-09-16T11:42:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror start mirror/laggy 2024-09-16T11:42:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:42:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:42:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:42:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/laggy 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:42:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:42:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:42:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/laggy 2024-09-16T11:42:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:42:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror start : start rbd mirror 2024-09-16T11:42:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:42:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:42:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror/laggy 2024-09-16T11:42:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:42:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:42:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:42:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:42:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:42:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:42:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:42:59.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:42:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:42:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:42:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror/laggy 2024-09-16T11:42:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:42:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-09-16T11:42:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:42:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:43:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:43:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:43:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:43:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:43:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:43:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:43:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:43:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:43:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='laggy: 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a6f1aea9-096b-4f04-b89b-59c015cd8d84 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: disconnected 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:01 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-09-16T11:43:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-09-16T11:43:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*disconnected' 2024-09-16T11:43:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: disconnected 2024-09-16T11:43:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:43:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:43:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:43:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:43:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:43:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:43:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:43:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:01' 2024-09-16T11:43:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-09-16 11:43:01' 2024-09-16T11:43:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:43:01' 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:43:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKlaggya6f1aea9-096b-4f04-b89b-59c015cd8d84up+errordisconnected58685871mirror.1smithi2042024-09-16 11:43:01cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:43:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:43:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2024-09-16T11:43:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:01' 2024-09-16T11:43:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2024-09-16T11:43:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-09-16T11:43:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:43:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-09-16T11:43:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:43:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - replay started after resync requested' 2024-09-16T11:43:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:43:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:43:02 - replay started after resync requested 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:43:02 - replay started after resync requested 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror laggy image_id 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16ef642a70c1'\''' 2024-09-16T11:43:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16ef642a70c1 2024-09-16T11:43:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-09-16T11:43:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16ef642a70c1 2024-09-16T11:43:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/laggy 2024-09-16T11:43:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-09-16T11:43:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 16ef642a70c1 2024-09-16T11:43:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:43:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef642a70c1 2024-09-16T11:43:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef642a70c1 2024-09-16T11:43:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:02.141 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16ef642a70c1 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef642a70c1 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16ef642a70c1 2024-09-16T11:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef642a70c1 2024-09-16T11:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16ef642a70c1 -o 16ef642a70c1 = 16ef642a70c1 2024-09-16T11:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16ef642a70c1 2024-09-16T11:43:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef642a70c1 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16ef642a70c1 2024-09-16T11:43:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef642a70c1 2024-09-16T11:43:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16ef642a70c1 -o 16ef642a70c1 = 16ef642a70c1 2024-09-16T11:43:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:43:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:43:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16ef642a70c1 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef642a70c1 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16ef642a70c1 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef642a70c1 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z 16ef642a70c1 -o 16ef642a70c1 = 16ef642a70c1 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = present 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16ef642a70c1 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef642a70c1 2024-09-16T11:43:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:43:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:43:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:43:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy present 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:43:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:43:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:43:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:43:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:43:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:43:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:43:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = deleted 2024-09-16T11:43:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:43:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:43:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16ef37c4b735 2024-09-16T11:43:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef37c4b735 2024-09-16T11:43:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 16ef37c4b735 2024-09-16T11:43:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:43:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:43:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:43:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:43:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:43:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:43:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:43:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:43:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:43:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:43:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:43:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:43:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:43:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:14.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 4 2024-09-16T11:43:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected] 2024-09-16T11:43:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 4 2024-09-16T11:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected] 2024-09-16T11:43:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' '!=' '' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' = '' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:43:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=8 2024-09-16T11:43:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:43:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=8 2024-09-16T11:43:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:43:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-09-16T11:43:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:43:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-09-16T11:43:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 8 = 8 -a 3 -le 1 2024-09-16T11:43:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:43:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 4 2024-09-16T11:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected] 2024-09-16T11:43:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 4 2024-09-16T11:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0], [object_number=27, tag_tid=6, entry_tid=39], [object_number=18, tag_tid=6, entry_tid=38]]], state=connected] 2024-09-16T11:43:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=8, entry_tid=1' '!=' 'object_number=25, tag_tid=8, entry_tid=1' 2024-09-16T11:43:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.4 2024-09-16T11:43:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' -a 'object_number=27, tag_tid=8, entry_tid=3' = 'object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2], [object_number=25, tag_tid=8, entry_tid=1], [object_number=24, tag_tid=8, entry_tid=0]]], state=connected] 2024-09-16T11:43:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=8, entry_tid=3' 2024-09-16T11:43:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror laggy 2024-09-16T11:43:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-laggy.export 2024-09-16T11:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-laggy.export 2024-09-16T11:43:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export 2024-09-16T11:43:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-laggy.export 2024-09-16T11:43:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-laggy.export 2024-09-16T11:43:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export 2024-09-16T11:43:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog ' - rbd_mirroring_resync_after_disconnect config option' 2024-09-16T11:43:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:43:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:43:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:43:15 - rbd_mirroring_resync_after_disconnect config option 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:43:15 - rbd_mirroring_resync_after_disconnect config option 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect true 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=true 2024-09-16T11:43:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect true 2024-09-16T11:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:16.353 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-09-16T11:43:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-09-16T11:43:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=25, tag_tid=9, entry_tid=1' 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-09-16T11:43:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=9, entry_tid=1], [object_number=24, tag_tid=9, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-09-16T11:43:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=9, entry_tid=1' 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=25, tag_tid=9, entry_tid=1' -a 'object_number=25, tag_tid=9, entry_tid=1' = 'object_number=25, tag_tid=9, entry_tid=1' 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id=16ef37c4b735 2024-09-16T11:43:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disconnect_image cluster2 mirror laggy 2024-09-16T11:43:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2024-09-16T11:43:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:client 88cac606-ed81-47bf-b993-eb31e1d47ba4 disconnected 2024-09-16T11:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy deleted 16ef37c4b735 2024-09-16T11:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=deleted 2024-09-16T11:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef37c4b735 2024-09-16T11:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16ef37c4b735 2024-09-16T11:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy deleted 16ef37c4b735 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=deleted 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id=16ef37c4b735 2024-09-16T11:43:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id= 2024-09-16T11:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test deleted = deleted 2024-09-16T11:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_present cluster1 mirror laggy present 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=present 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image laggy: (2) No such file or directory 2024-09-16T11:43:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image_id= 2024-09-16T11:43:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T11:43:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.1 2024-09-16T11:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_present cluster1 mirror laggy present '' 2024-09-16T11:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=present 2024-09-16T11:43:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id= 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=deleted 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_image_id 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror laggy 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/laggy 2024-09-16T11:43:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_image_id=16efe9ad95d7 2024-09-16T11:43:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 16efe9ad95d7 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' -o '' = 16efe9ad95d7 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=present 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test present = present 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_started cluster1 mirror laggy 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy started 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=started 2024-09-16T11:43:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy started 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=started 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:43:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:43:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:43:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:43:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:43:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:43:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 1 2024-09-16T11:43:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:43:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:43:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:43:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:43:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:43:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:43:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:43:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:43:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T11:43:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Replaying"' 2024-09-16T11:43:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:43:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: "state": "Replaying" 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ current_state=started 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test started = started 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:43:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-09-16T11:43:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0], [object_number=27, tag_tid=8, entry_tid=3], [object_number=26, tag_tid=8, entry_tid=2]]], state=connected] 2024-09-16T11:43:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=10, entry_tid=1' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' -a 'object_number=27, tag_tid=10, entry_tid=3' = 'object_number=25, tag_tid=10, entry_tid=1' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=10, entry_tid=1' '!=' '' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 'object_number=25, tag_tid=10, entry_tid=1' = '' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_mirror_pos='object_number=25, tag_tid=10, entry_tid=1' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:43:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_tag=10 2024-09-16T11:43:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=10, entry_tid=1' 2024-09-16T11:43:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-16T11:43:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_tag=10 2024-09-16T11:43:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:43:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_entry=3 2024-09-16T11:43:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo 'object_number=25, tag_tid=10, entry_tid=1' 2024-09-16T11:43:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-16T11:43:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cut -d= -f 2 2024-09-16T11:43:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_entry=1 2024-09-16T11:43:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 10 = 10 -a 3 -le 1 2024-09-16T11:43:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:43:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:43:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:20.205 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' -a 'object_number=27, tag_tid=10, entry_tid=3' = 'object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2], [object_number=25, tag_tid=10, entry_tid=1], [object_number=24, tag_tid=10, entry_tid=0]]], state=connected] 2024-09-16T11:43:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=27, tag_tid=10, entry_tid=3' 2024-09-16T11:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ compare_images mirror laggy 2024-09-16T11:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local ret=0 2024-09-16T11:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster2-mirror-laggy.export 2024-09-16T11:43:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster2-mirror-laggy.export 2024-09-16T11:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local rmt_export=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export 2024-09-16T11:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ mkfname cluster1-mirror-laggy.export 2024-09-16T11:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo cluster1-mirror-laggy.export 2024-09-16T11:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loc_export=/tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 export mirror/laggy /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export 2024-09-16T11:43:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 export mirror/laggy /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T11:43:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmp /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.export /tmp/tmp.MThcdnqr7J/cluster1-mirror-laggy.export 2024-09-16T11:43:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_image_meta cluster2 mirror laggy conf_rbd_mirroring_resync_after_disconnect false 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local key=conf_rbd_mirroring_resync_after_disconnect 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=false 2024-09-16T11:43:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 image-meta set mirror/laggy conf_rbd_mirroring_resync_after_disconnect false 2024-09-16T11:43:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:20.862 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' journal = journal ']' 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror laggy 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local local_cluster=cluster1 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ true 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T11:43:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0.2 2024-09-16T11:43:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ flush cluster1 mirror laggy 2024-09-16T11:43:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'cmd=rbd mirror flush' 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n laggy ']' 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cmd='rbd mirror flush mirror/laggy' 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/laggy 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_instance=cluster1 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance=cluster1 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local loop_instance 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/laggy 2024-09-16T11:43:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:43:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:0 2024-09-16T11:43:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:43:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror flush : flush rbd mirror 2024-09-16T11:43:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/laggy 2024-09-16T11:43:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:43:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster1:1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:43:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/laggy 2024-09-16T11:43:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_master_journal_position cluster2 mirror laggy 2024-09-16T11:43:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '' 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local id_regexp= 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-09-16T11:43:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-09-16T11:43:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ master_pos='object_number=25, tag_tid=11, entry_tid=1' 2024-09-16T11:43:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-09-16T11:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-09-16T11:43:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ mirror_pos='object_number=25, tag_tid=11, entry_tid=1' 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n 'object_number=25, tag_tid=11, entry_tid=1' -a 'object_number=25, tag_tid=11, entry_tid=1' = 'object_number=25, tag_tid=11, entry_tid=1' 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ disconnect_image cluster2 mirror laggy 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 journal client disconnect --image mirror/laggy 2024-09-16T11:43:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:client 88cac606-ed81-47bf-b993-eb31e1d47ba4 disconnected 2024-09-16T11:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_mirror_journal_position cluster2 mirror laggy 2024-09-16T11:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_journal_position cluster2 mirror laggy '..*' 2024-09-16T11:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=laggy 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local 'id_regexp=..*' 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ mkfname cluster2-mirror-laggy.status 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo cluster2-mirror-laggy.status 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-16T11:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local status_log=/tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster2 journal status --image mirror/laggy 2024-09-16T11:43:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ tee /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:minimum_set: 6 2024-09-16T11:43:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:active_set: 6 2024-09-16T11:43:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:registered clients: 2024-09-16T11:43:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=connected] 2024-09-16T11:43:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: [id=88cac606-ed81-47bf-b993-eb31e1d47ba4, commit_position=[positions=[[object_number=25, tag_tid=11, entry_tid=1], [object_number=24, tag_tid=11, entry_tid=0], [object_number=27, tag_tid=10, entry_tid=3], [object_number=26, tag_tid=10, entry_tid=2]]], state=disconnected] 2024-09-16T11:43:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.MThcdnqr7J/cluster2-mirror-laggy.status 2024-09-16T11:43:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -z '' 2024-09-16T11:43:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror laggy 2024-09-16T11:43:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:43:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror laggy stopped 2024-09-16T11:43:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:43:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:43:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:43:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/laggy 2024-09-16T11:43:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:43:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/laggy 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:43:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/laggy 2024-09-16T11:43:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:Invalid command: unused arguments: ['mirror/laggy'] 2024-09-16T11:43:22.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T11:43:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:admin_socket: invalid command 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/laggy 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:43:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:43:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:43:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/laggy 2024-09-16T11:43:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:43:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:43:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:43:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror laggy up+error disconnected 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-09-16T11:43:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror laggy up+error disconnected '' 2024-09-16T11:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=disconnected 2024-09-16T11:43:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/laggy 2024-09-16T11:43:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='laggy: 2024-09-16T11:43:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: a6f1aea9-096b-4f04-b89b-59c015cd8d84 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: disconnected 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.1 on smithi204 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:21 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-09-16T11:43:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-09-16T11:43:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*disconnected' 2024-09-16T11:43:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: disconnected 2024-09-16T11:43:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:43:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:43:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:43:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:43:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.1 on smithi204 2024-09-16T11:43:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:43:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:21' 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror laggy up+error '2024-09-16 11:43:21' 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:43:21' 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:43:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKlaggya6f1aea9-096b-4f04-b89b-59c015cd8d84up+errordisconnected58685871mirror.1smithi2042024-09-16 11:43:21cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:43:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:43:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/last_update' 2024-09-16T11:43:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:21' 2024-09-16T11:43:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''laggy'\'']/state' 2024-09-16T11:43:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-09-16T11:43:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:43:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-09-16T11:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror laggy 2024-09-16T11:43:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:43:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror laggy 2024-09-16T11:43:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=laggy 2024-09-16T11:43:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/laggy 2024-09-16T11:43:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/laggy 2024-09-16T11:43:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:43:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: split-brain' 2024-09-16T11:43:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:43:23.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:43:23 TEST: split-brain 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:43:23 TEST: split-brain 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ image=split-brain 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ create_image cluster2 mirror split-brain 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ shift 2024-09-16T11:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size=128 2024-09-16T11:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/split-brain 2024-09-16T11:43:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position 2024-09-16T11:43:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image split-brain: (2) No such file or directory 2024-09-16T11:43:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status= 2024-09-16T11:43:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:43:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+syncing 2024-09-16T11:43:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: bootstrapping, IMAGE_SYNC/COPY_IMAGE 2024-09-16T11:43:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:26 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:43:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:43:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:43:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:27 2024-09-16T11:43:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:43:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:43:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-16T11:43:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:43:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:43:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:43:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:43:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:43:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:27' 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2024-09-16 11:43:27' 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:43:27' 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:43:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKsplit-brain7c20ec84-6467-4461-b87c-38fd27c1139dup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}58615867mirror.0smithi2042024-09-16 11:43:27cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:43:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:43:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-09-16T11:43:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:27' 2024-09-16T11:43:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-09-16T11:43:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:43:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:43:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:43:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:43:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ promote_image cluster1 mirror split-brain --force 2024-09-16T11:43:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local force=--force 2024-09-16T11:43:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/split-brain --force 2024-09-16T11:43:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image promoted to primary 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_stopped cluster1 mirror split-brain 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_replay_state cluster1 mirror split-brain stopped 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state=stopped 2024-09-16T11:43:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local s 2024-09-16T11:43:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_image_replay_state cluster1 mirror split-brain stopped 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local test_state=stopped 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status_result 2024-09-16T11:43:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local current_state=stopped 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemons cluster1 rbd mirror status mirror/split-brain 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_instance=cluster1 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -i state 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance=cluster1 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local loop_instance 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sleep 0 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T11:43:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ seq 0 1 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/split-brain 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ shift 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:43:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local local_cluster=cluster1:0 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:43:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval local_cluster=cluster1 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ local_cluster=cluster1 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:43:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:43:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/split-brain 2024-09-16T11:43:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ return 0 2024-09-16T11:43:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status_result=' "state": "Stopped"' 2024-09-16T11:43:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo ' "state": "Stopped"' 2024-09-16T11:43:32.300 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -i Replaying 2024-09-16T11:43:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test stopped = stopped 2024-09-16T11:43:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+stopped 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+stopped '' '' 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern= 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:31 2024-09-16T11:43:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+stopped' 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+stopped 2024-09-16T11:43:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*' 2024-09-16T11:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: local image is primary 2024-09-16T11:43:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:43:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:43:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:43:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:43:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:43:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:43:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:43:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:31' 2024-09-16T11:43:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+stopped '2024-09-16 11:43:31' 2024-09-16T11:43:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+stopped 2024-09-16T11:43:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:43:31' 2024-09-16T11:43:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:43:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKsplit-brain7c20ec84-6467-4461-b87c-38fd27c1139dup+stoppedlocal image is primary58615867mirror.0smithi2042024-09-16 11:43:31cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:43:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:43:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-09-16T11:43:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:31' 2024-09-16T11:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-09-16T11:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+stopped 2024-09-16T11:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+stopped 2024-09-16T11:43:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+stopped 2024-09-16T11:43:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+stopped 2024-09-16T11:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ write_image cluster1 mirror split-brain 10 2024-09-16T11:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local count=10 2024-09-16T11:43:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local size= 2024-09-16T11:43:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ size=4096 2024-09-16T11:43:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster cluster1 bench mirror/split-brain --io-type write --io-size 4096 --io-threads 1 --io-total 40960 --io-pattern rand 2024-09-16T11:43:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:bench type write io_size 4096 io_threads 1 bytes 40960 pattern random 2024-09-16T11:43:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-16T11:43:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:elapsed: 0 ops: 10 ops/sec: 1428.59 bytes/sec: 5.6 MiB/s 2024-09-16T11:43:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ demote_image cluster1 mirror split-brain 2024-09-16T11:43:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/split-brain 2024-09-16T11:43:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Image demoted to non-primary 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+error split-brain 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:31 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-09-16T11:43:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+stopped 2024-09-16T11:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: local image is primary 2024-09-16T11:43:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:31 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-09-16T11:43:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:43:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+error split-brain '' 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=split-brain 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+error 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: split-brain detected 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:36 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+error' 2024-09-16T11:43:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+error 2024-09-16T11:43:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*split-brain' 2024-09-16T11:43:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: split-brain detected 2024-09-16T11:43:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:43:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:43:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:43:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:43:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:43:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:43:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:43:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:36' 2024-09-16T11:43:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+error '2024-09-16 11:43:36' 2024-09-16T11:43:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+error 2024-09-16T11:43:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:43:36' 2024-09-16T11:43:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:43:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKsplit-brain7c20ec84-6467-4461-b87c-38fd27c1139dup+errorsplit-brain detected58615867mirror.0smithi2042024-09-16 11:43:36cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:43:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:43:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-09-16T11:43:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:36' 2024-09-16T11:43:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-09-16T11:43:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+error 2024-09-16T11:43:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+error 2024-09-16T11:43:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+error 2024-09-16T11:43:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+error 2024-09-16T11:43:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ request_resync_image cluster1 mirror split-brain image_id 2024-09-16T11:43:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:40.798 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image_id_var_name=image_id 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ get_image_id cluster1 mirror split-brain 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local image=split-brain 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster=cluster1 info mirror/split-brain 2024-09-16T11:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T11:43:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'image_id='\''16ebcf67b6b5'\''' 2024-09-16T11:43:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ image_id=16ebcf67b6b5 2024-09-16T11:43:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval 'test -n "$image_id"' 2024-09-16T11:43:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ test -n 16ebcf67b6b5 2024-09-16T11:43:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster1 mirror image resync mirror/split-brain 2024-09-16T11:43:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:Flagged image for resync from primary 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: mirroring not enabled on the image 2024-09-16T11:43:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status= 2024-09-16T11:43:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:rbd: error opening image split-brain: (2) No such file or directory 2024-09-16T11:43:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status= 2024-09-16T11:43:43.979 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 4 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 1 2024-09-16T11:43:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T11:43:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 8 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_status_in_pool_dir cluster1 mirror split-brain up+replaying primary_position '' 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local description_pattern=primary_position 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local service_pattern= 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror image status mirror/split-brain 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='split-brain: 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: global_id: 7c20ec84-6467-4461-b87c-38fd27c1139d 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: up+replaying 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: service: mirror.0 on smithi204 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: 2024-09-16 11:43:48 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: peer_sites: 2024-09-16T11:43:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: name: cluster2 2024-09-16T11:43:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: state: down+unknown 2024-09-16T11:43:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: description: status not found 2024-09-16T11:43:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: last_update: ' 2024-09-16T11:43:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T11:43:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: state: up+replaying 2024-09-16T11:43:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^ description: .*primary_position' 2024-09-16T11:43:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2024-09-16T11:43:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n '' ']' 2024-09-16T11:43:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:43:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep '^up+' 2024-09-16T11:43:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:43:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T11:43:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout: service: mirror.0 on smithi204 2024-09-16T11:43:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update 2024-09-16T11:43:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T11:43:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:48' 2024-09-16T11:43:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test_mirror_pool_status_verbose cluster1 mirror split-brain up+replaying '2024-09-16 11:43:48' 2024-09-16T11:43:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local state_pattern=up+replaying 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local 'prev_last_update=2024-09-16 11:43:48' 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local status 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ CEPH_ARGS= 2024-09-16T11:43:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T11:43:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ status='WARNINGOKWARNING158615867mirror.0smithi20417.2.7-1643-gd75f2382trueOK58685871mirror.1smithi20417.2.7-1643-gd75f2382falseOKsplit-brain7c20ec84-6467-4461-b87c-38fd27c1139dup+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}}58615867mirror.0smithi2042024-09-16 11:43:48cluster2575d6853-c710-4ab7-a806-f3cf450240e2down+unknownstatus not found' 2024-09-16T11:43:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local last_update state 2024-09-16T11:43:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/last_update' 2024-09-16T11:43:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ last_update='2024-09-16 11:43:48' 2024-09-16T11:43:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''split-brain'\'']/state' 2024-09-16T11:43:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ state=up+replaying 2024-09-16T11:43:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo up+replaying 2024-09-16T11:43:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep up+replaying 2024-09-16T11:43:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:up+replaying 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image_retry cluster2 mirror split-brain 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T11:43:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ remove_image cluster2 mirror split-brain 2024-09-16T11:43:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local image=split-brain 2024-09-16T11:43:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 snap purge mirror/split-brain 2024-09-16T11:43:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd --cluster=cluster2 rm mirror/split-brain 2024-09-16T11:43:56.282 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T11:43:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-16T11:43:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:43:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:43:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:43:56 TEST: check if removed 'images'\''' OMAP are removed 2024-09-16T11:43:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:43:56 TEST: check if removed images' OMAP are removed 2024-09-16T11:43:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirrors cluster2 2024-09-16T11:43:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:43:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:0 2024-09-16T11:43:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:43:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:43:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:0 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=0 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=0 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:43:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:43:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ start_mirror cluster2:1 2024-09-16T11:43:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local val=cluster2:1 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster_var_name=cluster 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local instance_var_name=instance 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cluster=cluster2 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ instance=1 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval cluster=cluster2 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ eval instance=1 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:43:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:43:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T11:43:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-16T11:43:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:43:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-09-16T11:43:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:43:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-16T11:43:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T11:43:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-09-16T11:43:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:43:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:43:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-09-16T11:43:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-09-16T11:43:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-09-16T11:43:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:43:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:43:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 1 = 0 2024-09-16T11:43:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=1 2024-09-16T11:43:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 1 -eq 0 ']' 2024-09-16T11:43:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 2 2024-09-16T11:43:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:43:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:43:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=status_global 2024-09-16T11:43:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=status_global 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:43:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c status_global 2024-09-16T11:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirroring 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_ 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_ 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirroring 2024-09-16T11:43:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_ 2024-09-16T11:43:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T11:43:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:43:59.583 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pool=mirror 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local filter=image_map 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T11:43:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 0 2024-09-16T11:43:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:43:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local filter=image_map 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local pool=mirror 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T11:43:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ grep -c image_map 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test 0 = 0 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ error_code=0 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set -e 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ return 0 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ testlog 'TEST: no blocklists' 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ tee -a /tmp/tmp.MThcdnqr7J/rbd-mirror.test.log 2024-09-16T11:43:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ date '+%F %T' 2024-09-16T11:43:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo 2024-09-16 11:43:59 TEST: no blocklists 2024-09-16T11:43:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:2024-09-16 11:43:59 TEST: no blocklists 2024-09-16T11:43:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS='--id admin' 2024-09-16T11:43:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd blocklist ls 2024-09-16T11:43:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -q 'listed 0 entries' 2024-09-16T11:43:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS='--id admin' 2024-09-16T11:43:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd blocklist ls 2024-09-16T11:43:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ grep -q 'listed 0 entries' 2024-09-16T11:44:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ cleanup 0 2024-09-16T11:44:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local error_code=0 2024-09-16T11:44:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ set +e 2024-09-16T11:44:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T11:44:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z '' ']' 2024-09-16T11:44:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster instance 2024-09-16T11:44:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:44:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T11:44:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' does not exist 2024-09-16T11:44:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:44:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T11:44:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror' does not exist 2024-09-16T11:44:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:44:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T11:44:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' does not exist 2024-09-16T11:44:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ CEPH_ARGS= 2024-09-16T11:44:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T11:44:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:pool 'mirror_parent' does not exist 2024-09-16T11:44:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T11:44:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster1 2024-09-16T11:44:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1 2024-09-16T11:44:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:44:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:0 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:0 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:0 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:0 2024-09-16T11:44:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:44:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:44:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:44:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:44:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=68922 2024-09-16T11:44:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 68922 ']' 2024-09-16T11:44:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 68922 2024-09-16T11:44:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:44:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:44:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=68922 '$2 == pid {print; exit 1}' 2024-09-16T11:44:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:44:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=68922 '$2 == pid {print; exit 1}' 2024-09-16T11:44:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T11:44:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:44:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:44:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:44:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:44:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:44:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-16T11:44:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:0 2024-09-16T11:44:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:44:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T11:44:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T11:44:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:0 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:0 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:0 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:44:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:44:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T11:44:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:44:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:44:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:44:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T11:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T11:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T11:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster1:1 2024-09-16T11:44:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:44:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:44:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:44:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster1:1 2024-09-16T11:44:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster1:1 2024-09-16T11:44:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:44:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster1:1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:44:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=68950 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 68950 ']' 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 68950 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:44:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:44:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:04.953 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=68950 '$2 == pid {print; exit 1}' 2024-09-16T11:44:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:44:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=68950 '$2 == pid {print; exit 1}' 2024-09-16T11:44:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-16T11:44:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:44:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:44:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:04.989 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-16T11:44:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster1:1 2024-09-16T11:44:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:44:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T11:44:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T11:44:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster1:1 2024-09-16T11:44:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1:1 2024-09-16T11:44:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster1:1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-16T11:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:44:05.026 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-16T11:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T11:44:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirrors cluster2 2024-09-16T11:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2 2024-09-16T11:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:44:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ seq 0 1 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:0 '' 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:0 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:0 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:0 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:44:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:0 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=0 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=0 2024-09-16T11:44:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-16T11:44:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:44:05.045 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:44:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=72664 2024-09-16T11:44:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 72664 ']' 2024-09-16T11:44:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 72664 2024-09-16T11:44:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:44:05.047 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:44:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=72664 '$2 == pid {print; exit 1}' 2024-09-16T11:44:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:44:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=72664 '$2 == pid {print; exit 1}' 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:44:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:44:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:44:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T11:44:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:0 2024-09-16T11:44:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:44:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T11:44:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:44:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T11:44:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:0 2024-09-16T11:44:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:0 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:0 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=0 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=0 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=0 2024-09-16T11:44:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-16T11:44:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:44:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T11:44:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T11:44:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ stop_mirror cluster2:1 '' 2024-09-16T11:44:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local cluster=cluster2:1 2024-09-16T11:44:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local sig= 2024-09-16T11:44:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test -n '' 2024-09-16T11:44:06.124 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ local pid 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ daemon_pid_file cluster2:1 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster=cluster2:1 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local val=cluster2:1 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local cluster_var_name=cluster 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local instance_var_name=instance 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-16T11:44:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval cluster=cluster2 2024-09-16T11:44:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ cluster=cluster2 2024-09-16T11:44:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ eval instance=1 2024-09-16T11:44:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ instance=1 2024-09-16T11:44:06.127 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-16T11:44:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:44:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:44:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ pid=72699 2024-09-16T11:44:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -n 72699 ']' 2024-09-16T11:44:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ kill 72699 2024-09-16T11:44:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T11:44:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ sleep 1 2024-09-16T11:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:07.145 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=72699 '$2 == pid {print; exit 1}' 2024-09-16T11:44:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ break 2024-09-16T11:44:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ ps auxww 2024-09-16T11:44:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ awk -v pid=72699 '$2 == pid {print; exit 1}' 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster1 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:44:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:44:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:44:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:44:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-16T11:44:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local local_cluster=cluster2:1 2024-09-16T11:44:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2 2024-09-16T11:44:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-16T11:44:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=local_cluster 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval local_cluster=cluster2 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ local_cluster=cluster2 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-16T11:44:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:44:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ daemon_pid_file cluster2:1 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster=cluster2:1 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local val=cluster2:1 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local cluster_var_name=cluster 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ local instance_var_name=instance 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ cluster=cluster2 2024-09-16T11:44:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ instance=1 2024-09-16T11:44:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-16T11:44:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval cluster=cluster2 2024-09-16T11:44:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ cluster=cluster2 2024-09-16T11:44:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ eval instance=1 2024-09-16T11:44:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ instance=1 2024-09-16T11:44:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-16T11:44:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:44:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-16T11:44:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' -z 1 ']' 2024-09-16T11:44:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ test '' = /tmp/tmp.MThcdnqr7J 2024-09-16T11:44:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ rm -Rf /tmp/tmp.MThcdnqr7J 2024-09-16T11:44:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T11:44:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ echo OK 2024-09-16T11:44:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stdout:OK 2024-09-16T11:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi204.stderr:+ exit 0 2024-09-16T11:44:07.221 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:44:07.221 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-16T11:44:07.249 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_journal.sh'] on cluster1.client.mirror... 2024-09-16T11:44:07.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-16T11:44:07.562 DEBUG:teuthology.parallel:result is None 2024-09-16T11:44:07.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T11:44:07.587 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T11:44:07.587 DEBUG:teuthology.orchestra.run.smithi204:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T11:44:07.656 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T11:44:07.656 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T11:44:07.669 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2024-09-16T11:44:07.669 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:44:07.957 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:07.957 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":149,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:44:02.719810+0000","last_up_change":"2024-09-16T11:22:29.390391+0000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:30.726822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:33.891246+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":29226},{"type":"v1","addr":"172.21.15.204:6801","nonce":29226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":29226},{"type":"v1","addr":"172.21.15.204:6803","nonce":29226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":29226},{"type":"v1","addr":"172.21.15.204:6811","nonce":29226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":29226},{"type":"v1","addr":"172.21.15.204:6807","nonce":29226}]},"public_addr":"172.21.15.204:6801/29226","cluster_addr":"172.21.15.204:6803/29226","heartbeat_back_addr":"172.21.15.204:6811/29226","heartbeat_front_addr":"172.21.15.204:6807/29226","state":["exists","up"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":29227},{"type":"v1","addr":"172.21.15.204:6806","nonce":29227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":29227},{"type":"v1","addr":"172.21.15.204:6810","nonce":29227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":29227},{"type":"v1","addr":"172.21.15.204:6815","nonce":29227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":29227},{"type":"v1","addr":"172.21.15.204:6813","nonce":29227}]},"public_addr":"172.21.15.204:6806/29227","cluster_addr":"172.21.15.204:6810/29227","heartbeat_back_addr":"172.21.15.204:6815/29227","heartbeat_front_addr":"172.21.15.204:6813/29227","state":["exists","up"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":29228},{"type":"v1","addr":"172.21.15.204:6817","nonce":29228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":29228},{"type":"v1","addr":"172.21.15.204:6819","nonce":29228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":29228},{"type":"v1","addr":"172.21.15.204:6823","nonce":29228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":29228},{"type":"v1","addr":"172.21.15.204:6821","nonce":29228}]},"public_addr":"172.21.15.204:6817/29228","cluster_addr":"172.21.15.204:6819/29228","heartbeat_back_addr":"172.21.15.204:6823/29228","heartbeat_front_addr":"172.21.15.204:6821/29228","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.482114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.515043+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.540557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:44:07.969 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T11:44:07.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:44:08.272 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:08.273 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:44:08.284 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-16T11:44:07.025062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2397532,"kb_used_data":278644520,"kb_used_omap":32546,"kb_used_meta":0,"kb_avail":278641316,"statfs":{"total":287783780352,"available":285328707584,"internally_reserved":3280896,"allocated":285331988480,"data_stored":285331988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33327796,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1046,"num_objects":-27,"num_object_clones":0,"num_object_copies":-54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-27,"num_whiteouts":0,"num_read":-62512,"num_read_kb":-4150380,"num_write":-9350,"num_write_kb":-111439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-10035200,"num_keys_recovered":0,"num_objects_omap":-20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.724721+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:44:01.724721+0000","last_peered":"2024-09-16T11:44:01.724721+0000","last_clean":"2024-09-16T11:44:01.724721+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:44:01.724721+0000","last_undegraded":"2024-09-16T11:44:01.724721+0000","last_fullsized":"2024-09-16T11:44:01.724721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728143+0000","last_change":"2024-09-16T11:22:37.638530+0000","last_active":"2024-09-16T11:44:01.728143+0000","last_peered":"2024-09-16T11:44:01.728143+0000","last_clean":"2024-09-16T11:44:01.728143+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:44:01.728143+0000","last_undegraded":"2024-09-16T11:44:01.728143+0000","last_fullsized":"2024-09-16T11:44:01.728143+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:18.738957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.727144+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:44:01.727144+0000","last_peered":"2024-09-16T11:44:01.727144+0000","last_clean":"2024-09-16T11:44:01.727144+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:44:01.727144+0000","last_undegraded":"2024-09-16T11:44:01.727144+0000","last_fullsized":"2024-09-16T11:44:01.727144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:56.608460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.722037+0000","last_change":"2024-09-16T11:22:37.638272+0000","last_active":"2024-09-16T11:44:01.722037+0000","last_peered":"2024-09-16T11:44:01.722037+0000","last_clean":"2024-09-16T11:44:01.722037+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:44:01.722037+0000","last_undegraded":"2024-09-16T11:44:01.722037+0000","last_fullsized":"2024-09-16T11:44:01.722037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:34.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":148,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-16T11:43:55.718323+0000","last_change":"2024-09-16T11:22:37.635551+0000","last_active":"2024-09-16T11:43:55.718323+0000","last_peered":"2024-09-16T11:43:55.718323+0000","last_clean":"2024-09-16T11:43:55.718323+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:43:55.718323+0000","last_undegraded":"2024-09-16T11:43:55.718323+0000","last_fullsized":"2024-09-16T11:43:55.718323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:04.960663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.321736+0000","last_change":"2024-09-16T11:22:38.486357+0000","last_active":"2024-09-16T11:44:02.321736+0000","last_peered":"2024-09-16T11:44:02.321736+0000","last_clean":"2024-09-16T11:44:02.321736+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:44:02.321736+0000","last_undegraded":"2024-09-16T11:44:02.321736+0000","last_fullsized":"2024-09-16T11:44:02.321736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:39.481341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.317756+0000","last_change":"2024-09-16T11:22:38.486359+0000","last_active":"2024-09-16T11:44:02.317756+0000","last_peered":"2024-09-16T11:44:02.317756+0000","last_clean":"2024-09-16T11:44:02.317756+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:44:02.317756+0000","last_undegraded":"2024-09-16T11:44:02.317756+0000","last_fullsized":"2024-09-16T11:44:02.317756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:54.166730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.726935+0000","last_change":"2024-09-16T11:22:37.638406+0000","last_active":"2024-09-16T11:44:01.726935+0000","last_peered":"2024-09-16T11:44:01.726935+0000","last_clean":"2024-09-16T11:44:01.726935+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:44:01.726935+0000","last_undegraded":"2024-09-16T11:44:01.726935+0000","last_fullsized":"2024-09-16T11:44:01.726935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:42.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728151+0000","last_change":"2024-09-16T11:22:32.471966+0000","last_active":"2024-09-16T11:44:01.728151+0000","last_peered":"2024-09-16T11:44:01.728151+0000","last_clean":"2024-09-16T11:44:01.728151+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:44:01.728151+0000","last_undegraded":"2024-09-16T11:44:01.728151+0000","last_fullsized":"2024-09-16T11:44:01.728151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_clean_scrub_stamp":"2024-09-16T11:22:31.398024+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:25.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738629,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798808,"kb_used_data":92881812,"kb_used_omap":10583,"kb_used_meta":0,"kb_avail":92880808,"statfs":{"total":95927926784,"available":95109947392,"internally_reserved":1028096,"allocated":95110975488,"data_stored":95110975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10837384,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.30199999999999999,"15min":0.28799999999999998},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.14000000000000001},"max":{"1min":0.45500000000000002,"5min":0.48199999999999998,"15min":0.52500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.17199999999999999},"max":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.52200000000000002},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.153},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.48499999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.313},"min":{"1min":0.22,"5min":0.22,"15min":0.17199999999999999},"max":{"1min":0.5,"5min":0.50600000000000001,"15min":0.53700000000000003},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738629,"num_pgs":98,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799276,"kb_used_data":92881484,"kb_used_omap":10239,"kb_used_meta":0,"kb_avail":92880340,"statfs":{"total":95927926784,"available":95109468160,"internally_reserved":1171456,"allocated":95110639616,"data_stored":95110639616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10485126,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.48199999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.28100000000000003,"15min":0.28599999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.158},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.49299999999999999},"last":0.22}]},{"osd":2,"last update":"Mon Sep 16 11:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":0.29499999999999998,"15min":0.307},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.5720000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.30599999999999999,"15min":0.313},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.505,"5min":0.505,"15min":2.452},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":93,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799448,"kb_used_data":92881224,"kb_used_omap":11723,"kb_used_meta":0,"kb_avail":92880168,"statfs":{"total":95927926784,"available":95109292032,"internally_reserved":1081344,"allocated":95110373376,"data_stored":95110373376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005286,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29199999999999998,"15min":0.28599999999999998},"min":{"1min":0.19400000000000001,"5min":0.16,"15min":0.14499999999999999},"max":{"1min":0.379,"5min":0.42999999999999999,"15min":1.327},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.317,"15min":0.29499999999999998},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.16400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.90200000000000002},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.308,"15min":0.29399999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17399999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33400000000000002,"15min":0.315},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":1.3740000000000001},"last":0.223}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:08.285 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:44:08.541 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:08.542 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:44:08.553 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-16T11:44:07.025062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2397532,"kb_used_data":278644520,"kb_used_omap":32546,"kb_used_meta":0,"kb_avail":278641316,"statfs":{"total":287783780352,"available":285328707584,"internally_reserved":3280896,"allocated":285331988480,"data_stored":285331988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33327796,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1046,"num_objects":-27,"num_object_clones":0,"num_object_copies":-54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-27,"num_whiteouts":0,"num_read":-62512,"num_read_kb":-4150380,"num_write":-9350,"num_write_kb":-111439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-10035200,"num_keys_recovered":0,"num_objects_omap":-20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.724721+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:44:01.724721+0000","last_peered":"2024-09-16T11:44:01.724721+0000","last_clean":"2024-09-16T11:44:01.724721+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:44:01.724721+0000","last_undegraded":"2024-09-16T11:44:01.724721+0000","last_fullsized":"2024-09-16T11:44:01.724721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728143+0000","last_change":"2024-09-16T11:22:37.638530+0000","last_active":"2024-09-16T11:44:01.728143+0000","last_peered":"2024-09-16T11:44:01.728143+0000","last_clean":"2024-09-16T11:44:01.728143+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:44:01.728143+0000","last_undegraded":"2024-09-16T11:44:01.728143+0000","last_fullsized":"2024-09-16T11:44:01.728143+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:18.738957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.727144+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:44:01.727144+0000","last_peered":"2024-09-16T11:44:01.727144+0000","last_clean":"2024-09-16T11:44:01.727144+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:44:01.727144+0000","last_undegraded":"2024-09-16T11:44:01.727144+0000","last_fullsized":"2024-09-16T11:44:01.727144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:56.608460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.722037+0000","last_change":"2024-09-16T11:22:37.638272+0000","last_active":"2024-09-16T11:44:01.722037+0000","last_peered":"2024-09-16T11:44:01.722037+0000","last_clean":"2024-09-16T11:44:01.722037+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:44:01.722037+0000","last_undegraded":"2024-09-16T11:44:01.722037+0000","last_fullsized":"2024-09-16T11:44:01.722037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:34.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":148,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-16T11:43:55.718323+0000","last_change":"2024-09-16T11:22:37.635551+0000","last_active":"2024-09-16T11:43:55.718323+0000","last_peered":"2024-09-16T11:43:55.718323+0000","last_clean":"2024-09-16T11:43:55.718323+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:43:55.718323+0000","last_undegraded":"2024-09-16T11:43:55.718323+0000","last_fullsized":"2024-09-16T11:43:55.718323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:04.960663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.321736+0000","last_change":"2024-09-16T11:22:38.486357+0000","last_active":"2024-09-16T11:44:02.321736+0000","last_peered":"2024-09-16T11:44:02.321736+0000","last_clean":"2024-09-16T11:44:02.321736+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:44:02.321736+0000","last_undegraded":"2024-09-16T11:44:02.321736+0000","last_fullsized":"2024-09-16T11:44:02.321736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:39.481341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.317756+0000","last_change":"2024-09-16T11:22:38.486359+0000","last_active":"2024-09-16T11:44:02.317756+0000","last_peered":"2024-09-16T11:44:02.317756+0000","last_clean":"2024-09-16T11:44:02.317756+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:44:02.317756+0000","last_undegraded":"2024-09-16T11:44:02.317756+0000","last_fullsized":"2024-09-16T11:44:02.317756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:54.166730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.726935+0000","last_change":"2024-09-16T11:22:37.638406+0000","last_active":"2024-09-16T11:44:01.726935+0000","last_peered":"2024-09-16T11:44:01.726935+0000","last_clean":"2024-09-16T11:44:01.726935+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:44:01.726935+0000","last_undegraded":"2024-09-16T11:44:01.726935+0000","last_fullsized":"2024-09-16T11:44:01.726935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:42.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728151+0000","last_change":"2024-09-16T11:22:32.471966+0000","last_active":"2024-09-16T11:44:01.728151+0000","last_peered":"2024-09-16T11:44:01.728151+0000","last_clean":"2024-09-16T11:44:01.728151+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:44:01.728151+0000","last_undegraded":"2024-09-16T11:44:01.728151+0000","last_fullsized":"2024-09-16T11:44:01.728151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_clean_scrub_stamp":"2024-09-16T11:22:31.398024+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:25.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738629,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798808,"kb_used_data":92881812,"kb_used_omap":10583,"kb_used_meta":0,"kb_avail":92880808,"statfs":{"total":95927926784,"available":95109947392,"internally_reserved":1028096,"allocated":95110975488,"data_stored":95110975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10837384,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.30199999999999999,"15min":0.28799999999999998},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.14000000000000001},"max":{"1min":0.45500000000000002,"5min":0.48199999999999998,"15min":0.52500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.17199999999999999},"max":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.52200000000000002},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.153},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.48499999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.313},"min":{"1min":0.22,"5min":0.22,"15min":0.17199999999999999},"max":{"1min":0.5,"5min":0.50600000000000001,"15min":0.53700000000000003},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738629,"num_pgs":98,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799276,"kb_used_data":92881484,"kb_used_omap":10239,"kb_used_meta":0,"kb_avail":92880340,"statfs":{"total":95927926784,"available":95109468160,"internally_reserved":1171456,"allocated":95110639616,"data_stored":95110639616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10485126,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.48199999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.28100000000000003,"15min":0.28599999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.158},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.49299999999999999},"last":0.22}]},{"osd":2,"last update":"Mon Sep 16 11:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":0.29499999999999998,"15min":0.307},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.5720000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.30599999999999999,"15min":0.313},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.505,"5min":0.505,"15min":2.452},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":93,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799448,"kb_used_data":92881224,"kb_used_omap":11723,"kb_used_meta":0,"kb_avail":92880168,"statfs":{"total":95927926784,"available":95109292032,"internally_reserved":1081344,"allocated":95110373376,"data_stored":95110373376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005286,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29199999999999998,"15min":0.28599999999999998},"min":{"1min":0.19400000000000001,"5min":0.16,"15min":0.14499999999999999},"max":{"1min":0.379,"5min":0.42999999999999999,"15min":1.327},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.317,"15min":0.29499999999999998},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.16400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.90200000000000002},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.308,"15min":0.29399999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17399999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33400000000000002,"15min":0.315},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":1.3740000000000001},"last":0.223}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:08.554 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T11:44:08.555 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:44:08.814 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:08.814 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:44:08.826 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-16T11:44:07.025062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2397532,"kb_used_data":278644520,"kb_used_omap":32546,"kb_used_meta":0,"kb_avail":278641316,"statfs":{"total":287783780352,"available":285328707584,"internally_reserved":3280896,"allocated":285331988480,"data_stored":285331988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33327796,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1046,"num_objects":-27,"num_object_clones":0,"num_object_copies":-54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-27,"num_whiteouts":0,"num_read":-62512,"num_read_kb":-4150380,"num_write":-9350,"num_write_kb":-111439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-10035200,"num_keys_recovered":0,"num_objects_omap":-20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001916"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.724721+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:44:01.724721+0000","last_peered":"2024-09-16T11:44:01.724721+0000","last_clean":"2024-09-16T11:44:01.724721+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:44:01.724721+0000","last_undegraded":"2024-09-16T11:44:01.724721+0000","last_fullsized":"2024-09-16T11:44:01.724721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728143+0000","last_change":"2024-09-16T11:22:37.638530+0000","last_active":"2024-09-16T11:44:01.728143+0000","last_peered":"2024-09-16T11:44:01.728143+0000","last_clean":"2024-09-16T11:44:01.728143+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:44:01.728143+0000","last_undegraded":"2024-09-16T11:44:01.728143+0000","last_fullsized":"2024-09-16T11:44:01.728143+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:18.738957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.727144+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:44:01.727144+0000","last_peered":"2024-09-16T11:44:01.727144+0000","last_clean":"2024-09-16T11:44:01.727144+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:44:01.727144+0000","last_undegraded":"2024-09-16T11:44:01.727144+0000","last_fullsized":"2024-09-16T11:44:01.727144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:56.608460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.722037+0000","last_change":"2024-09-16T11:22:37.638272+0000","last_active":"2024-09-16T11:44:01.722037+0000","last_peered":"2024-09-16T11:44:01.722037+0000","last_clean":"2024-09-16T11:44:01.722037+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:44:01.722037+0000","last_undegraded":"2024-09-16T11:44:01.722037+0000","last_fullsized":"2024-09-16T11:44:01.722037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:34.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":148,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-16T11:43:55.718323+0000","last_change":"2024-09-16T11:22:37.635551+0000","last_active":"2024-09-16T11:43:55.718323+0000","last_peered":"2024-09-16T11:43:55.718323+0000","last_clean":"2024-09-16T11:43:55.718323+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:43:55.718323+0000","last_undegraded":"2024-09-16T11:43:55.718323+0000","last_fullsized":"2024-09-16T11:43:55.718323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:04.960663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.321736+0000","last_change":"2024-09-16T11:22:38.486357+0000","last_active":"2024-09-16T11:44:02.321736+0000","last_peered":"2024-09-16T11:44:02.321736+0000","last_clean":"2024-09-16T11:44:02.321736+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:44:02.321736+0000","last_undegraded":"2024-09-16T11:44:02.321736+0000","last_fullsized":"2024-09-16T11:44:02.321736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:39.481341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.317756+0000","last_change":"2024-09-16T11:22:38.486359+0000","last_active":"2024-09-16T11:44:02.317756+0000","last_peered":"2024-09-16T11:44:02.317756+0000","last_clean":"2024-09-16T11:44:02.317756+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:44:02.317756+0000","last_undegraded":"2024-09-16T11:44:02.317756+0000","last_fullsized":"2024-09-16T11:44:02.317756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:54.166730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.726935+0000","last_change":"2024-09-16T11:22:37.638406+0000","last_active":"2024-09-16T11:44:01.726935+0000","last_peered":"2024-09-16T11:44:01.726935+0000","last_clean":"2024-09-16T11:44:01.726935+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:44:01.726935+0000","last_undegraded":"2024-09-16T11:44:01.726935+0000","last_fullsized":"2024-09-16T11:44:01.726935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:42.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728151+0000","last_change":"2024-09-16T11:22:32.471966+0000","last_active":"2024-09-16T11:44:01.728151+0000","last_peered":"2024-09-16T11:44:01.728151+0000","last_clean":"2024-09-16T11:44:01.728151+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:44:01.728151+0000","last_undegraded":"2024-09-16T11:44:01.728151+0000","last_fullsized":"2024-09-16T11:44:01.728151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_clean_scrub_stamp":"2024-09-16T11:22:31.398024+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:25.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738629,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798808,"kb_used_data":92881812,"kb_used_omap":10583,"kb_used_meta":0,"kb_avail":92880808,"statfs":{"total":95927926784,"available":95109947392,"internally_reserved":1028096,"allocated":95110975488,"data_stored":95110975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10837384,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.30199999999999999,"15min":0.28799999999999998},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.14000000000000001},"max":{"1min":0.45500000000000002,"5min":0.48199999999999998,"15min":0.52500000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.17199999999999999},"max":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.52200000000000002},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.153},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.48499999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.313},"min":{"1min":0.22,"5min":0.22,"15min":0.17199999999999999},"max":{"1min":0.5,"5min":0.50600000000000001,"15min":0.53700000000000003},"last":0.33100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738629,"num_pgs":98,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799276,"kb_used_data":92881484,"kb_used_omap":10239,"kb_used_meta":0,"kb_avail":92880340,"statfs":{"total":95927926784,"available":95109468160,"internally_reserved":1171456,"allocated":95110639616,"data_stored":95110639616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10485126,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.48199999999999998},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.28100000000000003,"15min":0.28599999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.158},"max":{"1min":0.46800000000000003,"5min":0.47999999999999998,"15min":0.49299999999999999},"last":0.22}]},{"osd":2,"last update":"Mon Sep 16 11:43:04 2024","interfaces":[{"interface":"back","average":{"1min":0.28000000000000003,"5min":0.29499999999999998,"15min":0.307},"min":{"1min":0.17499999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":2.5720000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.30599999999999999,"15min":0.313},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.505,"5min":0.505,"15min":2.452},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":93,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799448,"kb_used_data":92881224,"kb_used_omap":11723,"kb_used_meta":0,"kb_avail":92880168,"statfs":{"total":95927926784,"available":95109292032,"internally_reserved":1081344,"allocated":95110373376,"data_stored":95110373376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12005286,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.29199999999999998,"15min":0.28599999999999998},"min":{"1min":0.19400000000000001,"5min":0.16,"15min":0.14499999999999999},"max":{"1min":0.379,"5min":0.42999999999999999,"15min":1.327},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.317,"15min":0.29499999999999998},"min":{"1min":0.216,"5min":0.21099999999999999,"15min":0.16400000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.90200000000000002},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.308,"15min":0.29399999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17399999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33400000000000002,"15min":0.315},"min":{"1min":0.20599999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":1.3740000000000001},"last":0.223}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:08.827 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T11:44:09.086 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:09.086 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":149,"fsid":"8fbb69d7-ce8e-4174-822f-8bd0c8eba787","created":"2024-09-16T11:22:25.579813+0000","modified":"2024-09-16T11:44:02.719810+0000","last_up_change":"2024-09-16T11:22:29.390391+0000","last_in_change":"2024-09-16T11:22:26.388141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:30.726822+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:33.891246+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f01f38ee-93c2-4c2c-a776-70b9fb002a04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":29226},{"type":"v1","addr":"172.21.15.204:6801","nonce":29226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":29226},{"type":"v1","addr":"172.21.15.204:6803","nonce":29226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":29226},{"type":"v1","addr":"172.21.15.204:6811","nonce":29226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":29226},{"type":"v1","addr":"172.21.15.204:6807","nonce":29226}]},"public_addr":"172.21.15.204:6801/29226","cluster_addr":"172.21.15.204:6803/29226","heartbeat_back_addr":"172.21.15.204:6811/29226","heartbeat_front_addr":"172.21.15.204:6807/29226","state":["exists","up"]},{"osd":1,"uuid":"2b6c4ac5-2edf-4df3-974a-7f37ce290945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":29227},{"type":"v1","addr":"172.21.15.204:6806","nonce":29227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":29227},{"type":"v1","addr":"172.21.15.204:6810","nonce":29227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":29227},{"type":"v1","addr":"172.21.15.204:6815","nonce":29227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":29227},{"type":"v1","addr":"172.21.15.204:6813","nonce":29227}]},"public_addr":"172.21.15.204:6806/29227","cluster_addr":"172.21.15.204:6810/29227","heartbeat_back_addr":"172.21.15.204:6815/29227","heartbeat_front_addr":"172.21.15.204:6813/29227","state":["exists","up"]},{"osd":2,"uuid":"c21f06cf-02c3-4106-82d1-d9f01e320a2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":29228},{"type":"v1","addr":"172.21.15.204:6817","nonce":29228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":29228},{"type":"v1","addr":"172.21.15.204:6819","nonce":29228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":29228},{"type":"v1","addr":"172.21.15.204:6823","nonce":29228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":29228},{"type":"v1","addr":"172.21.15.204:6821","nonce":29228}]},"public_addr":"172.21.15.204:6817/29228","cluster_addr":"172.21.15.204:6819/29228","heartbeat_back_addr":"172.21.15.204:6823/29228","heartbeat_front_addr":"172.21.15.204:6821/29228","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.482114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.515043+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:27.540557+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:44:10.100 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-16T11:44:10.101 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:44:10.229 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T11:44:10.230 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:44:10.230 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T11:44:10.240 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-16T11:44:10.500 INFO:teuthology.orchestra.run.smithi204.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T11:44:10.513 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-16T11:44:10.513 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:44:10.627 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T11:44:10.627 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:44:10.627 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T11:44:10.638 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-16T11:44:10.897 INFO:teuthology.orchestra.run.smithi204.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T11:44:10.909 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-16T11:44:10.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:44:11.024 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-16T11:44:11.024 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:44:11.024 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-16T11:44:11.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-16T11:44:11.293 INFO:teuthology.orchestra.run.smithi204.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T11:44:11.306 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:44:11.562 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:11.563 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:44:11.575 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-09-16T11:44:11.025730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393388,"kb_used_data":278645460,"kb_used_omap":32803,"kb_used_meta":0,"kb_avail":278645460,"statfs":{"total":287783780352,"available":285332951040,"internally_reserved":0,"allocated":285332951040,"data_stored":285332951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33590708,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-7,"num_object_clones":0,"num_object_copies":-14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":-4113,"num_read_kb":-600387,"num_write":-940,"num_write_kb":-2292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.305359"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.724721+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:44:01.724721+0000","last_peered":"2024-09-16T11:44:01.724721+0000","last_clean":"2024-09-16T11:44:01.724721+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:44:01.724721+0000","last_undegraded":"2024-09-16T11:44:01.724721+0000","last_fullsized":"2024-09-16T11:44:01.724721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.167208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728143+0000","last_change":"2024-09-16T11:22:37.638530+0000","last_active":"2024-09-16T11:44:01.728143+0000","last_peered":"2024-09-16T11:44:01.728143+0000","last_clean":"2024-09-16T11:44:01.728143+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:44:01.728143+0000","last_undegraded":"2024-09-16T11:44:01.728143+0000","last_fullsized":"2024-09-16T11:44:01.728143+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:18.738957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.727144+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:44:01.727144+0000","last_peered":"2024-09-16T11:44:01.727144+0000","last_clean":"2024-09-16T11:44:01.727144+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:44:01.727144+0000","last_undegraded":"2024-09-16T11:44:01.727144+0000","last_fullsized":"2024-09-16T11:44:01.727144+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:00:56.608460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":147,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.722037+0000","last_change":"2024-09-16T11:22:37.638272+0000","last_active":"2024-09-16T11:44:01.722037+0000","last_peered":"2024-09-16T11:44:01.722037+0000","last_clean":"2024-09-16T11:44:01.722037+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:44:01.722037+0000","last_undegraded":"2024-09-16T11:44:01.722037+0000","last_fullsized":"2024-09-16T11:44:01.722037+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:20:34.154248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":148,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-09-16T11:43:55.718323+0000","last_change":"2024-09-16T11:22:37.635551+0000","last_active":"2024-09-16T11:43:55.718323+0000","last_peered":"2024-09-16T11:43:55.718323+0000","last_clean":"2024-09-16T11:43:55.718323+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:43:55.718323+0000","last_undegraded":"2024-09-16T11:43:55.718323+0000","last_fullsized":"2024-09-16T11:43:55.718323+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:23:04.960663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.321736+0000","last_change":"2024-09-16T11:22:38.486357+0000","last_active":"2024-09-16T11:44:02.321736+0000","last_peered":"2024-09-16T11:44:02.321736+0000","last_clean":"2024-09-16T11:44:02.321736+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:44:02.321736+0000","last_undegraded":"2024-09-16T11:44:02.321736+0000","last_fullsized":"2024-09-16T11:44:02.321736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:56:39.481341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.317756+0000","last_change":"2024-09-16T11:22:38.486359+0000","last_active":"2024-09-16T11:44:02.317756+0000","last_peered":"2024-09-16T11:44:02.317756+0000","last_clean":"2024-09-16T11:44:02.317756+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:44:02.317756+0000","last_undegraded":"2024-09-16T11:44:02.317756+0000","last_fullsized":"2024-09-16T11:44:02.317756+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:54.166730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":148,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.726935+0000","last_change":"2024-09-16T11:22:37.638406+0000","last_active":"2024-09-16T11:44:01.726935+0000","last_peered":"2024-09-16T11:44:01.726935+0000","last_clean":"2024-09-16T11:44:01.726935+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:44:01.726935+0000","last_undegraded":"2024-09-16T11:44:01.726935+0000","last_fullsized":"2024-09-16T11:44:01.726935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:42.492283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.728151+0000","last_change":"2024-09-16T11:22:32.471966+0000","last_active":"2024-09-16T11:44:01.728151+0000","last_peered":"2024-09-16T11:44:01.728151+0000","last_clean":"2024-09-16T11:44:01.728151+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:44:01.728151+0000","last_undegraded":"2024-09-16T11:44:01.728151+0000","last_fullsized":"2024-09-16T11:44:01.728151+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:31.398024+0000","last_clean_scrub_stamp":"2024-09-16T11:22:31.398024+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:19:25.186625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738630,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797404,"kb_used_data":92882212,"kb_used_omap":10601,"kb_used_meta":0,"kb_avail":92882212,"statfs":{"total":95927926784,"available":95111385088,"internally_reserved":0,"allocated":95111385088,"data_stored":95111385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10856343,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.30199999999999999,"15min":0.28799999999999998},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.14000000000000001},"max":{"1min":0.45500000000000002,"5min":0.48199999999999998,"15min":0.52500000000000002},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.17199999999999999},"max":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.52200000000000002},"last":0.28899999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:43:12 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.153},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.48499999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.313},"min":{"1min":0.22,"5min":0.22,"15min":0.17199999999999999},"max":{"1min":0.5,"5min":0.50600000000000001,"15min":0.53700000000000003},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738630,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797992,"kb_used_data":92881624,"kb_used_omap":10239,"kb_used_meta":0,"kb_avail":92881624,"statfs":{"total":95927926784,"available":95110782976,"internally_reserved":0,"allocated":95110782976,"data_stored":95110782976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10485176,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28000000000000003},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.436,"5min":0.44700000000000001,"15min":0.48199999999999998},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.28499999999999998,"15min":0.28599999999999998},"min":{"1min":0.189,"5min":0.182,"15min":0.158},"max":{"1min":0.41599999999999998,"5min":0.47999999999999998,"15min":0.49299999999999999},"last":0.216}]},{"osd":2,"last update":"Mon Sep 16 11:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.30599999999999999},"min":{"1min":0.17699999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.39300000000000002,"5min":0.49099999999999999,"15min":2.5720000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.312},"min":{"1min":0.161,"5min":0.158,"15min":0.158},"max":{"1min":0.41399999999999998,"5min":0.505,"15min":2.452},"last":0.25800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738630,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797992,"kb_used_data":92881624,"kb_used_omap":11962,"kb_used_meta":0,"kb_avail":92881624,"statfs":{"total":95927926784,"available":95110782976,"internally_reserved":0,"allocated":95110782976,"data_stored":95110782976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12249189,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:09 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.187,"5min":0.16,"15min":0.158},"max":{"1min":0.34000000000000002,"5min":0.42999999999999999,"15min":1.327},"last":0.192},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.313,"15min":0.29499999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.16400000000000001},"max":{"1min":0.36699999999999999,"5min":0.69799999999999995,"15min":0.90200000000000002},"last":0.214}]},{"osd":2,"last update":"Mon Sep 16 11:44:09 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.308,"15min":0.29499999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.17399999999999999},"max":{"1min":0.38600000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.33300000000000002,"15min":0.316},"min":{"1min":0.223,"5min":0.20599999999999999,"15min":0.17199999999999999},"max":{"1min":0.40500000000000003,"5min":0.72699999999999998,"15min":1.3740000000000001},"last":0.246}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:11.578 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.578 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.578 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T11:22:31.398024+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:11.579 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:44:31.581 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:44:31.855 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:31.855 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:44:31.866 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-09-16T11:44:31.029284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393432,"kb_used_data":278645460,"kb_used_omap":32826,"kb_used_meta":0,"kb_avail":278645416,"statfs":{"total":287783780352,"available":285332905984,"internally_reserved":45056,"allocated":285332951040,"data_stored":285332951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33614450,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002145"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":148,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.725370+0000","last_change":"2024-09-16T11:22:37.638744+0000","last_active":"2024-09-16T11:44:02.725370+0000","last_peered":"2024-09-16T11:44:02.725370+0000","last_clean":"2024-09-16T11:44:02.725370+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:44:02.725370+0000","last_undegraded":"2024-09-16T11:44:02.725370+0000","last_fullsized":"2024-09-16T11:44:02.725370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:29.625408+0000","last_change":"2024-09-16T11:44:29.625381+0000","last_active":"2024-09-16T11:44:29.625408+0000","last_peered":"2024-09-16T11:44:29.625408+0000","last_clean":"2024-09-16T11:44:29.625408+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:44:29.625408+0000","last_undegraded":"2024-09-16T11:44:29.625408+0000","last_fullsized":"2024-09-16T11:44:29.625408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:29.625323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:29.625323+0000","last_clean_scrub_stamp":"2024-09-16T11:44:29.625323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:47:53.388220+0000","scrub_duration":0.001272785,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":148,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.725397+0000","last_change":"2024-09-16T11:22:37.638783+0000","last_active":"2024-09-16T11:44:02.725397+0000","last_peered":"2024-09-16T11:44:02.725397+0000","last_clean":"2024-09-16T11:44:02.725397+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:44:02.725397+0000","last_undegraded":"2024-09-16T11:44:02.725397+0000","last_fullsized":"2024-09-16T11:44:02.725397+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:34.615622+0000","last_clean_scrub_stamp":"2024-09-16T11:22:34.615622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:27.621060+0000","last_change":"2024-09-16T11:44:27.621034+0000","last_active":"2024-09-16T11:44:27.621060+0000","last_peered":"2024-09-16T11:44:27.621060+0000","last_clean":"2024-09-16T11:44:27.621060+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:44:27.621060+0000","last_undegraded":"2024-09-16T11:44:27.621060+0000","last_fullsized":"2024-09-16T11:44:27.621060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:27.620970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:27.620970+0000","last_clean_scrub_stamp":"2024-09-16T11:44:27.620970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:19.233481+0000","scrub_duration":0.0014356250000000001,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"149'3","reported_seq":160,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:16.323109+0000","last_change":"2024-09-16T11:44:16.323084+0000","last_active":"2024-09-16T11:44:16.323109+0000","last_peered":"2024-09-16T11:44:16.323109+0000","last_clean":"2024-09-16T11:44:16.323109+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:44:16.323109+0000","last_undegraded":"2024-09-16T11:44:16.323109+0000","last_fullsized":"2024-09-16T11:44:16.323109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'3","last_scrub_stamp":"2024-09-16T11:44:16.323025+0000","last_deep_scrub":"149'3","last_deep_scrub_stamp":"2024-09-16T11:44:16.323025+0000","last_clean_scrub_stamp":"2024-09-16T11:44:16.323025+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:34.720284+0000","scrub_duration":0.0025390780000000002,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":159,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:23.757539+0000","last_change":"2024-09-16T11:44:23.757515+0000","last_active":"2024-09-16T11:44:23.757539+0000","last_peered":"2024-09-16T11:44:23.757539+0000","last_clean":"2024-09-16T11:44:23.757539+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:44:23.757539+0000","last_undegraded":"2024-09-16T11:44:23.757539+0000","last_fullsized":"2024-09-16T11:44:23.757539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:23.757459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:23.757459+0000","last_clean_scrub_stamp":"2024-09-16T11:44:23.757459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:22.086862+0000","scrub_duration":0.0013576339999999999,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:27.783119+0000","last_change":"2024-09-16T11:44:27.783092+0000","last_active":"2024-09-16T11:44:27.783119+0000","last_peered":"2024-09-16T11:44:27.783119+0000","last_clean":"2024-09-16T11:44:27.783119+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:44:27.783119+0000","last_undegraded":"2024-09-16T11:44:27.783119+0000","last_fullsized":"2024-09-16T11:44:27.783119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:27.783034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:27.783034+0000","last_clean_scrub_stamp":"2024-09-16T11:44:27.783034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:53.977614+0000","scrub_duration":0.0012839080000000001,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"149'2","reported_seq":160,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:28.627633+0000","last_change":"2024-09-16T11:44:28.627607+0000","last_active":"2024-09-16T11:44:28.627633+0000","last_peered":"2024-09-16T11:44:28.627633+0000","last_clean":"2024-09-16T11:44:28.627633+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:44:28.627633+0000","last_undegraded":"2024-09-16T11:44:28.627633+0000","last_fullsized":"2024-09-16T11:44:28.627633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'2","last_scrub_stamp":"2024-09-16T11:44:28.627547+0000","last_deep_scrub":"149'2","last_deep_scrub_stamp":"2024-09-16T11:44:28.627547+0000","last_clean_scrub_stamp":"2024-09-16T11:44:28.627547+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:50:19.141986+0000","scrub_duration":0.0022165209999999999,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"149'54","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:23.550304+0000","last_change":"2024-09-16T11:44:23.550279+0000","last_active":"2024-09-16T11:44:23.550304+0000","last_peered":"2024-09-16T11:44:23.550304+0000","last_clean":"2024-09-16T11:44:23.550304+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:44:23.550304+0000","last_undegraded":"2024-09-16T11:44:23.550304+0000","last_fullsized":"2024-09-16T11:44:23.550304+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'54","last_scrub_stamp":"2024-09-16T11:44:23.550219+0000","last_deep_scrub":"149'54","last_deep_scrub_stamp":"2024-09-16T11:44:23.550219+0000","last_clean_scrub_stamp":"2024-09-16T11:44:23.550219+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:40:20.810993+0000","scrub_duration":0.0044504619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738634,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797420,"kb_used_data":92882212,"kb_used_omap":10606,"kb_used_meta":0,"kb_avail":92882196,"statfs":{"total":95927926784,"available":95111368704,"internally_reserved":16384,"allocated":95111385088,"data_stored":95111385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10861286,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:12 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.30099999999999999,"15min":0.28999999999999998},"min":{"1min":0.189,"5min":0.16400000000000001,"15min":0.14000000000000001},"max":{"1min":0.39200000000000002,"5min":0.48199999999999998,"15min":0.52500000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33800000000000002,"15min":0.32000000000000001},"min":{"1min":0.24199999999999999,"5min":0.22500000000000001,"15min":0.17199999999999999},"max":{"1min":0.45200000000000001,"5min":0.48199999999999998,"15min":0.52200000000000002},"last":0.47899999999999998}]},{"osd":1,"last update":"Mon Sep 16 11:44:12 2024","interfaces":[{"interface":"back","average":{"1min":0.28100000000000003,"5min":0.29399999999999998,"15min":0.29099999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.153},"max":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.318,"5min":0.32900000000000001,"15min":0.314},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17199999999999999},"max":{"1min":0.45400000000000001,"5min":0.5,"15min":0.53700000000000003},"last":0.44700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738634,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798016,"kb_used_data":92881624,"kb_used_omap":10250,"kb_used_meta":0,"kb_avail":92881600,"statfs":{"total":95927926784,"available":95110758400,"internally_reserved":24576,"allocated":95110782976,"data_stored":95110782976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10497022,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28000000000000003},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.436,"5min":0.44700000000000001,"15min":0.48199999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.28499999999999998,"15min":0.28599999999999998},"min":{"1min":0.189,"5min":0.182,"15min":0.158},"max":{"1min":0.41599999999999998,"5min":0.47999999999999998,"15min":0.49299999999999999},"last":0.37}]},{"osd":2,"last update":"Mon Sep 16 11:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.30599999999999999},"min":{"1min":0.17699999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.39300000000000002,"5min":0.49099999999999999,"15min":2.5720000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.312},"min":{"1min":0.161,"5min":0.158,"15min":0.158},"max":{"1min":0.41399999999999998,"5min":0.505,"15min":2.452},"last":0.41799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738634,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797996,"kb_used_data":92881624,"kb_used_omap":11968,"kb_used_meta":0,"kb_avail":92881620,"statfs":{"total":95927926784,"available":95110778880,"internally_reserved":4096,"allocated":95110782976,"data_stored":95110782976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12256142,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:09 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.187,"5min":0.16,"15min":0.158},"max":{"1min":0.34000000000000002,"5min":0.42999999999999999,"15min":1.327},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.313,"15min":0.29499999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.16400000000000001},"max":{"1min":0.36699999999999999,"5min":0.69799999999999995,"15min":0.90200000000000002},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:44:09 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.308,"15min":0.29499999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.17399999999999999},"max":{"1min":0.38600000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.33300000000000002,"15min":0.316},"min":{"1min":0.223,"5min":0.20599999999999999,"15min":0.17199999999999999},"max":{"1min":0.40500000000000003,"5min":0.72699999999999998,"15min":1.3740000000000001},"last":0.31900000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:31.867 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:31.868 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T11:22:34.615622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:31.868 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:44:51.869 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T11:44:52.140 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-16T11:44:52.141 INFO:teuthology.orchestra.run.smithi204.stderr:dumped all 2024-09-16T11:44:52.152 INFO:teuthology.orchestra.run.smithi204.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-09-16T11:44:51.033684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393388,"kb_used_data":278645460,"kb_used_omap":32833,"kb_used_meta":0,"kb_avail":278645460,"statfs":{"total":287783780352,"available":285332951040,"internally_reserved":0,"allocated":285332951040,"data_stored":285332951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33621921,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002537"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:33.582394+0000","last_change":"2024-09-16T11:44:33.582367+0000","last_active":"2024-09-16T11:44:33.582394+0000","last_peered":"2024-09-16T11:44:33.582394+0000","last_clean":"2024-09-16T11:44:33.582394+0000","last_became_active":"2024-09-16T11:22:35.628107+0000","last_became_peered":"2024-09-16T11:22:35.628107+0000","last_unstale":"2024-09-16T11:44:33.582394+0000","last_undegraded":"2024-09-16T11:44:33.582394+0000","last_fullsized":"2024-09-16T11:44:33.582394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:33.582308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:33.582308+0000","last_clean_scrub_stamp":"2024-09-16T11:44:33.582308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:32.039072+0000","scrub_duration":0.001383085,"objects_trimmed":0,"snaptrim_duration":0.00060603000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:29.625408+0000","last_change":"2024-09-16T11:44:29.625381+0000","last_active":"2024-09-16T11:44:29.625408+0000","last_peered":"2024-09-16T11:44:29.625408+0000","last_clean":"2024-09-16T11:44:29.625408+0000","last_became_active":"2024-09-16T11:22:35.628019+0000","last_became_peered":"2024-09-16T11:22:35.628019+0000","last_unstale":"2024-09-16T11:44:29.625408+0000","last_undegraded":"2024-09-16T11:44:29.625408+0000","last_fullsized":"2024-09-16T11:44:29.625408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:29.625323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:29.625323+0000","last_clean_scrub_stamp":"2024-09-16T11:44:29.625323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:47:53.388220+0000","scrub_duration":0.001272785,"objects_trimmed":0,"snaptrim_duration":0.00048767400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:30.614995+0000","last_change":"2024-09-16T11:44:30.614969+0000","last_active":"2024-09-16T11:44:30.614995+0000","last_peered":"2024-09-16T11:44:30.614995+0000","last_clean":"2024-09-16T11:44:30.614995+0000","last_became_active":"2024-09-16T11:22:35.627880+0000","last_became_peered":"2024-09-16T11:22:35.627880+0000","last_unstale":"2024-09-16T11:44:30.614995+0000","last_undegraded":"2024-09-16T11:44:30.614995+0000","last_fullsized":"2024-09-16T11:44:30.614995+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:30.614911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:30.614911+0000","last_clean_scrub_stamp":"2024-09-16T11:44:30.614911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:52:01.529603+0000","scrub_duration":0.0013223899999999999,"objects_trimmed":0,"snaptrim_duration":0.00064353499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":158,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:27.621060+0000","last_change":"2024-09-16T11:44:27.621034+0000","last_active":"2024-09-16T11:44:27.621060+0000","last_peered":"2024-09-16T11:44:27.621060+0000","last_clean":"2024-09-16T11:44:27.621060+0000","last_became_active":"2024-09-16T11:22:35.627533+0000","last_became_peered":"2024-09-16T11:22:35.627533+0000","last_unstale":"2024-09-16T11:44:27.621060+0000","last_undegraded":"2024-09-16T11:44:27.621060+0000","last_fullsized":"2024-09-16T11:44:27.621060+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:27.620970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:27.620970+0000","last_clean_scrub_stamp":"2024-09-16T11:44:27.620970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:54:19.233481+0000","scrub_duration":0.0014356250000000001,"objects_trimmed":0,"snaptrim_duration":0.00035921599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"149'3","reported_seq":160,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:16.323109+0000","last_change":"2024-09-16T11:44:16.323084+0000","last_active":"2024-09-16T11:44:16.323109+0000","last_peered":"2024-09-16T11:44:16.323109+0000","last_clean":"2024-09-16T11:44:16.323109+0000","last_became_active":"2024-09-16T11:22:35.626803+0000","last_became_peered":"2024-09-16T11:22:35.626803+0000","last_unstale":"2024-09-16T11:44:16.323109+0000","last_undegraded":"2024-09-16T11:44:16.323109+0000","last_fullsized":"2024-09-16T11:44:16.323109+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'3","last_scrub_stamp":"2024-09-16T11:44:16.323025+0000","last_deep_scrub":"149'3","last_deep_scrub_stamp":"2024-09-16T11:44:16.323025+0000","last_clean_scrub_stamp":"2024-09-16T11:44:16.323025+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:34.720284+0000","scrub_duration":0.0025390780000000002,"objects_trimmed":0,"snaptrim_duration":0.000328685,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":159,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:23.757539+0000","last_change":"2024-09-16T11:44:23.757515+0000","last_active":"2024-09-16T11:44:23.757539+0000","last_peered":"2024-09-16T11:44:23.757539+0000","last_clean":"2024-09-16T11:44:23.757539+0000","last_became_active":"2024-09-16T11:22:35.626107+0000","last_became_peered":"2024-09-16T11:22:35.626107+0000","last_unstale":"2024-09-16T11:44:23.757539+0000","last_undegraded":"2024-09-16T11:44:23.757539+0000","last_fullsized":"2024-09-16T11:44:23.757539+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:23.757459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:23.757459+0000","last_clean_scrub_stamp":"2024-09-16T11:44:23.757459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:18:22.086862+0000","scrub_duration":0.0013576339999999999,"objects_trimmed":0,"snaptrim_duration":0.00027275999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":159,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:27.783119+0000","last_change":"2024-09-16T11:44:27.783092+0000","last_active":"2024-09-16T11:44:27.783119+0000","last_peered":"2024-09-16T11:44:27.783119+0000","last_clean":"2024-09-16T11:44:27.783119+0000","last_became_active":"2024-09-16T11:22:35.625595+0000","last_became_peered":"2024-09-16T11:22:35.625595+0000","last_unstale":"2024-09-16T11:44:27.783119+0000","last_undegraded":"2024-09-16T11:44:27.783119+0000","last_fullsized":"2024-09-16T11:44:27.783119+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:27.783034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:27.783034+0000","last_clean_scrub_stamp":"2024-09-16T11:44:27.783034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:53.977614+0000","scrub_duration":0.0012839080000000001,"objects_trimmed":0,"snaptrim_duration":0.00025904699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"149'2","reported_seq":160,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:28.627633+0000","last_change":"2024-09-16T11:44:28.627607+0000","last_active":"2024-09-16T11:44:28.627633+0000","last_peered":"2024-09-16T11:44:28.627633+0000","last_clean":"2024-09-16T11:44:28.627633+0000","last_became_active":"2024-09-16T11:22:35.625970+0000","last_became_peered":"2024-09-16T11:22:35.625970+0000","last_unstale":"2024-09-16T11:44:28.627633+0000","last_undegraded":"2024-09-16T11:44:28.627633+0000","last_fullsized":"2024-09-16T11:44:28.627633+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'2","last_scrub_stamp":"2024-09-16T11:44:28.627547+0000","last_deep_scrub":"149'2","last_deep_scrub_stamp":"2024-09-16T11:44:28.627547+0000","last_clean_scrub_stamp":"2024-09-16T11:44:28.627547+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:50:19.141986+0000","scrub_duration":0.0022165209999999999,"objects_trimmed":0,"snaptrim_duration":0.00039409500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"149'54","reported_seq":226,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-09-16T11:44:23.550304+0000","last_change":"2024-09-16T11:44:23.550279+0000","last_active":"2024-09-16T11:44:23.550304+0000","last_peered":"2024-09-16T11:44:23.550304+0000","last_clean":"2024-09-16T11:44:23.550304+0000","last_became_active":"2024-09-16T11:22:32.471156+0000","last_became_peered":"2024-09-16T11:22:32.471156+0000","last_unstale":"2024-09-16T11:44:23.550304+0000","last_undegraded":"2024-09-16T11:44:23.550304+0000","last_fullsized":"2024-09-16T11:44:23.550304+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'54","last_scrub_stamp":"2024-09-16T11:44:23.550219+0000","last_deep_scrub":"149'54","last_deep_scrub_stamp":"2024-09-16T11:44:23.550219+0000","last_clean_scrub_stamp":"2024-09-16T11:44:23.550219+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:40:20.810993+0000","scrub_duration":0.0044504619999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797404,"kb_used_data":92882212,"kb_used_omap":10606,"kb_used_meta":0,"kb_avail":92882212,"statfs":{"total":95927926784,"available":95111385088,"internally_reserved":0,"allocated":95111385088,"data_stored":95111385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10861336,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:12 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.30099999999999999,"15min":0.28999999999999998},"min":{"1min":0.189,"5min":0.16400000000000001,"15min":0.14000000000000001},"max":{"1min":0.39200000000000002,"5min":0.48199999999999998,"15min":0.52500000000000002},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33800000000000002,"15min":0.32000000000000001},"min":{"1min":0.24199999999999999,"5min":0.22500000000000001,"15min":0.17199999999999999},"max":{"1min":0.45200000000000001,"5min":0.48199999999999998,"15min":0.52200000000000002},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Sep 16 11:44:12 2024","interfaces":[{"interface":"back","average":{"1min":0.28100000000000003,"5min":0.29399999999999998,"15min":0.29099999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.153},"max":{"1min":0.40600000000000003,"5min":0.46500000000000002,"15min":0.48499999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.318,"5min":0.32900000000000001,"15min":0.314},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.17199999999999999},"max":{"1min":0.45400000000000001,"5min":0.5,"15min":0.53700000000000003},"last":0.32400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738638,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797992,"kb_used_data":92881624,"kb_used_omap":10254,"kb_used_meta":0,"kb_avail":92881624,"statfs":{"total":95927926784,"available":95110782976,"internally_reserved":0,"allocated":95110782976,"data_stored":95110782976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10500623,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28000000000000003},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.436,"5min":0.44700000000000001,"15min":0.48199999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.28499999999999998,"15min":0.28599999999999998},"min":{"1min":0.189,"5min":0.182,"15min":0.158},"max":{"1min":0.41599999999999998,"5min":0.47999999999999998,"15min":0.49299999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:05 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.29799999999999999,"15min":0.30599999999999999},"min":{"1min":0.17699999999999999,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.39300000000000002,"5min":0.49099999999999999,"15min":2.5720000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.311,"5min":0.30499999999999999,"15min":0.312},"min":{"1min":0.161,"5min":0.158,"15min":0.158},"max":{"1min":0.41399999999999998,"5min":0.505,"15min":2.452},"last":0.34000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738638,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797992,"kb_used_data":92881624,"kb_used_omap":11972,"kb_used_meta":0,"kb_avail":92881624,"statfs":{"total":95927926784,"available":95110782976,"internally_reserved":0,"allocated":95110782976,"data_stored":95110782976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12259962,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:09 2024","interfaces":[{"interface":"back","average":{"1min":0.27200000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"min":{"1min":0.187,"5min":0.16,"15min":0.158},"max":{"1min":0.34000000000000002,"5min":0.42999999999999999,"15min":1.327},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.313,"15min":0.29499999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.16400000000000001},"max":{"1min":0.36699999999999999,"5min":0.69799999999999995,"15min":0.90200000000000002},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Sep 16 11:44:09 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.308,"15min":0.29499999999999998},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.17399999999999999},"max":{"1min":0.38600000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.33300000000000002,"15min":0.316},"min":{"1min":0.223,"5min":0.20599999999999999,"15min":0.17199999999999999},"max":{"1min":0.40500000000000003,"5min":0.72699999999999998,"15min":1.3740000000000001},"last":0.39000000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:52.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-16T11:44:52.426 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T11:44:52.426 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T11:44:52.426 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-16T11:44:52.426 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:44:52.440 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-16T11:44:52.441 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-16T11:44:52.441 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:44:52.495 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-16T11:44:52.496 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-16T11:44:52.496 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:44:52.549 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-16T11:44:52.549 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T11:44:52.550 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-16T11:44:52.550 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:44:52.579 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-16T11:44:52.579 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T11:44:52.579 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-16T11:44:52.579 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:44:52.601 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-16T11:44:52.601 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T11:44:52.601 DEBUG:teuthology.orchestra.run.smithi204:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T11:44:52.669 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:44:52.669 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-16T11:44:52.853 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:44:52.853 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-16T11:44:52.943 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:44:52.943 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-16T11:44:53.010 INFO:tasks.ceph:Archiving mon data... 2024-09-16T11:44:53.011 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/data/cluster2.mon.a.tgz 2024-09-16T11:44:53.011 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-16T11:44:53.026 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.MRD6KjQp72 2024-09-16T11:44:53.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.MRD6KjQp72 2024-09-16T11:44:53.187 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.MRD6KjQp72 2024-09-16T11:44:53.270 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.MRD6KjQp72 is 429KB 2024-09-16T11:44:53.367 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.MRD6KjQp72 2024-09-16T11:44:53.422 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T11:44:53.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:44:53.424 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:44:53.487 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T11:44:53.496 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2024-09-16T11:44:53.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:44:53.779 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:44:53.780 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":154,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:44:02.203477+0000","last_up_change":"2024-09-16T11:22:06.056486+0000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:08.131565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:10.731960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":28635},{"type":"v1","addr":"172.21.15.53:6809","nonce":28635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":28635},{"type":"v1","addr":"172.21.15.53:6811","nonce":28635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":28635},{"type":"v1","addr":"172.21.15.53:6815","nonce":28635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":28635},{"type":"v1","addr":"172.21.15.53:6813","nonce":28635}]},"public_addr":"172.21.15.53:6809/28635","cluster_addr":"172.21.15.53:6811/28635","heartbeat_back_addr":"172.21.15.53:6815/28635","heartbeat_front_addr":"172.21.15.53:6813/28635","state":["exists","up"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":28626},{"type":"v1","addr":"172.21.15.53:6801","nonce":28626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":28626},{"type":"v1","addr":"172.21.15.53:6803","nonce":28626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":28626},{"type":"v1","addr":"172.21.15.53:6807","nonce":28626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":28626},{"type":"v1","addr":"172.21.15.53:6805","nonce":28626}]},"public_addr":"172.21.15.53:6801/28626","cluster_addr":"172.21.15.53:6803/28626","heartbeat_back_addr":"172.21.15.53:6807/28626","heartbeat_front_addr":"172.21.15.53:6805/28626","state":["exists","up"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":28638},{"type":"v1","addr":"172.21.15.53:6817","nonce":28638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":28638},{"type":"v1","addr":"172.21.15.53:6819","nonce":28638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":28638},{"type":"v1","addr":"172.21.15.53:6823","nonce":28638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":28638},{"type":"v1","addr":"172.21.15.53:6821","nonce":28638}]},"public_addr":"172.21.15.53:6817/28638","cluster_addr":"172.21.15.53:6819/28638","heartbeat_back_addr":"172.21.15.53:6823/28638","heartbeat_front_addr":"172.21.15.53:6821/28638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107422+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.097413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":1},{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:44:53.792 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T11:44:53.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:44:54.062 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:44:54.063 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:44:54.074 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":832,"stamp":"2024-09-16T11:44:52.437056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393496,"kb_used_data":278645352,"kb_used_omap":33825,"kb_used_meta":0,"kb_avail":278645352,"statfs":{"total":287783780352,"available":285332840448,"internally_reserved":0,"allocated":285332840448,"data_stored":285332840448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34637729,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002428"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.968952+0000","last_change":"2024-09-16T11:22:14.120245+0000","last_active":"2024-09-16T11:44:01.968952+0000","last_peered":"2024-09-16T11:44:01.968952+0000","last_clean":"2024-09-16T11:44:01.968952+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:44:01.968952+0000","last_undegraded":"2024-09-16T11:44:01.968952+0000","last_fullsized":"2024-09-16T11:44:01.968952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:50.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.971612+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:44:01.971612+0000","last_peered":"2024-09-16T11:44:01.971612+0000","last_clean":"2024-09-16T11:44:01.971612+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:44:01.971612+0000","last_undegraded":"2024-09-16T11:44:01.971612+0000","last_fullsized":"2024-09-16T11:44:01.971612+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:07.810548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.970998+0000","last_change":"2024-09-16T11:22:14.120010+0000","last_active":"2024-09-16T11:44:01.970998+0000","last_peered":"2024-09-16T11:44:01.970998+0000","last_clean":"2024-09-16T11:44:01.970998+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:44:01.970998+0000","last_undegraded":"2024-09-16T11:44:01.970998+0000","last_fullsized":"2024-09-16T11:44:01.970998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:43.889549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.969535+0000","last_change":"2024-09-16T11:22:14.119525+0000","last_active":"2024-09-16T11:44:01.969535+0000","last_peered":"2024-09-16T11:44:01.969535+0000","last_clean":"2024-09-16T11:44:01.969535+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:44:01.969535+0000","last_undegraded":"2024-09-16T11:44:01.969535+0000","last_fullsized":"2024-09-16T11:44:01.969535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:21.893309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":156,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.205119+0000","last_change":"2024-09-16T11:22:14.118460+0000","last_active":"2024-09-16T11:44:01.205119+0000","last_peered":"2024-09-16T11:44:01.205119+0000","last_clean":"2024-09-16T11:44:01.205119+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:44:01.205119+0000","last_undegraded":"2024-09-16T11:44:01.205119+0000","last_fullsized":"2024-09-16T11:44:01.205119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:29:26.098123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.548248+0000","last_change":"2024-09-16T11:22:14.116075+0000","last_active":"2024-09-16T11:44:01.548248+0000","last_peered":"2024-09-16T11:44:01.548248+0000","last_clean":"2024-09-16T11:44:01.548248+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:44:01.548248+0000","last_undegraded":"2024-09-16T11:44:01.548248+0000","last_fullsized":"2024-09-16T11:44:01.548248+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:59.034856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.550791+0000","last_change":"2024-09-16T11:22:14.116152+0000","last_active":"2024-09-16T11:44:01.550791+0000","last_peered":"2024-09-16T11:44:01.550791+0000","last_clean":"2024-09-16T11:44:01.550791+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:44:01.550791+0000","last_undegraded":"2024-09-16T11:44:01.550791+0000","last_fullsized":"2024-09-16T11:44:01.550791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:26.664156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":153,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.972784+0000","last_change":"2024-09-16T11:22:14.119568+0000","last_active":"2024-09-16T11:44:01.972784+0000","last_peered":"2024-09-16T11:44:01.972784+0000","last_clean":"2024-09-16T11:44:01.972784+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:44:01.972784+0000","last_undegraded":"2024-09-16T11:44:01.972784+0000","last_fullsized":"2024-09-16T11:44:01.972784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:46.653548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.973330+0000","last_change":"2024-09-16T11:22:10.960454+0000","last_active":"2024-09-16T11:44:01.973330+0000","last_peered":"2024-09-16T11:44:01.973330+0000","last_clean":"2024-09-16T11:44:01.973330+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:44:01.973330+0000","last_undegraded":"2024-09-16T11:44:01.973330+0000","last_fullsized":"2024-09-16T11:44:01.973330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_clean_scrub_stamp":"2024-09-16T11:22:09.097683+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:09.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738644,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797440,"kb_used_data":92882176,"kb_used_omap":10619,"kb_used_meta":0,"kb_avail":92882176,"statfs":{"total":95927926784,"available":95111348224,"internally_reserved":0,"allocated":95111348224,"data_stored":95111348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10874854,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.316,"15min":0.311},"min":{"1min":0.23100000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.33500000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.18099999999999999},"max":{"1min":0.47599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.317,"15min":0.32600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.51500000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738643,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":10885,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11146572,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.307},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.44700000000000001,"5min":0.497,"15min":0.497},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46400000000000002},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.19800000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.49199999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.189},"max":{"1min":0.441,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":12320,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12616303,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.30599999999999999},"min":{"1min":0.245,"5min":0.221,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.22,"15min":0.214},"max":{"1min":0.46000000000000002,"5min":0.48799999999999999,"15min":1.9279999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":0.436,"5min":1.026,"15min":1.026},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35299999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":0.495,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.25800000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:54.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:44:54.343 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:44:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:44:54.355 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":832,"stamp":"2024-09-16T11:44:52.437056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393496,"kb_used_data":278645352,"kb_used_omap":33825,"kb_used_meta":0,"kb_avail":278645352,"statfs":{"total":287783780352,"available":285332840448,"internally_reserved":0,"allocated":285332840448,"data_stored":285332840448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34637729,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002428"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.968952+0000","last_change":"2024-09-16T11:22:14.120245+0000","last_active":"2024-09-16T11:44:01.968952+0000","last_peered":"2024-09-16T11:44:01.968952+0000","last_clean":"2024-09-16T11:44:01.968952+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:44:01.968952+0000","last_undegraded":"2024-09-16T11:44:01.968952+0000","last_fullsized":"2024-09-16T11:44:01.968952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:50.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.971612+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:44:01.971612+0000","last_peered":"2024-09-16T11:44:01.971612+0000","last_clean":"2024-09-16T11:44:01.971612+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:44:01.971612+0000","last_undegraded":"2024-09-16T11:44:01.971612+0000","last_fullsized":"2024-09-16T11:44:01.971612+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:07.810548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.970998+0000","last_change":"2024-09-16T11:22:14.120010+0000","last_active":"2024-09-16T11:44:01.970998+0000","last_peered":"2024-09-16T11:44:01.970998+0000","last_clean":"2024-09-16T11:44:01.970998+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:44:01.970998+0000","last_undegraded":"2024-09-16T11:44:01.970998+0000","last_fullsized":"2024-09-16T11:44:01.970998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:43.889549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.969535+0000","last_change":"2024-09-16T11:22:14.119525+0000","last_active":"2024-09-16T11:44:01.969535+0000","last_peered":"2024-09-16T11:44:01.969535+0000","last_clean":"2024-09-16T11:44:01.969535+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:44:01.969535+0000","last_undegraded":"2024-09-16T11:44:01.969535+0000","last_fullsized":"2024-09-16T11:44:01.969535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:21.893309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":156,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.205119+0000","last_change":"2024-09-16T11:22:14.118460+0000","last_active":"2024-09-16T11:44:01.205119+0000","last_peered":"2024-09-16T11:44:01.205119+0000","last_clean":"2024-09-16T11:44:01.205119+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:44:01.205119+0000","last_undegraded":"2024-09-16T11:44:01.205119+0000","last_fullsized":"2024-09-16T11:44:01.205119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:29:26.098123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.548248+0000","last_change":"2024-09-16T11:22:14.116075+0000","last_active":"2024-09-16T11:44:01.548248+0000","last_peered":"2024-09-16T11:44:01.548248+0000","last_clean":"2024-09-16T11:44:01.548248+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:44:01.548248+0000","last_undegraded":"2024-09-16T11:44:01.548248+0000","last_fullsized":"2024-09-16T11:44:01.548248+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:59.034856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.550791+0000","last_change":"2024-09-16T11:22:14.116152+0000","last_active":"2024-09-16T11:44:01.550791+0000","last_peered":"2024-09-16T11:44:01.550791+0000","last_clean":"2024-09-16T11:44:01.550791+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:44:01.550791+0000","last_undegraded":"2024-09-16T11:44:01.550791+0000","last_fullsized":"2024-09-16T11:44:01.550791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:26.664156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":153,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.972784+0000","last_change":"2024-09-16T11:22:14.119568+0000","last_active":"2024-09-16T11:44:01.972784+0000","last_peered":"2024-09-16T11:44:01.972784+0000","last_clean":"2024-09-16T11:44:01.972784+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:44:01.972784+0000","last_undegraded":"2024-09-16T11:44:01.972784+0000","last_fullsized":"2024-09-16T11:44:01.972784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:46.653548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.973330+0000","last_change":"2024-09-16T11:22:10.960454+0000","last_active":"2024-09-16T11:44:01.973330+0000","last_peered":"2024-09-16T11:44:01.973330+0000","last_clean":"2024-09-16T11:44:01.973330+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:44:01.973330+0000","last_undegraded":"2024-09-16T11:44:01.973330+0000","last_fullsized":"2024-09-16T11:44:01.973330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_clean_scrub_stamp":"2024-09-16T11:22:09.097683+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:09.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738644,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797440,"kb_used_data":92882176,"kb_used_omap":10619,"kb_used_meta":0,"kb_avail":92882176,"statfs":{"total":95927926784,"available":95111348224,"internally_reserved":0,"allocated":95111348224,"data_stored":95111348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10874854,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.316,"15min":0.311},"min":{"1min":0.23100000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.33500000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.18099999999999999},"max":{"1min":0.47599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.317,"15min":0.32600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.51500000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738643,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":10885,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11146572,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.307},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.44700000000000001,"5min":0.497,"15min":0.497},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46400000000000002},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.19800000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.49199999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.189},"max":{"1min":0.441,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":12320,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12616303,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.30599999999999999},"min":{"1min":0.245,"5min":0.221,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.22,"15min":0.214},"max":{"1min":0.46000000000000002,"5min":0.48799999999999999,"15min":1.9279999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":0.436,"5min":1.026,"15min":1.026},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35299999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":0.495,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.25800000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:54.357 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T11:44:54.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:44:54.626 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:44:54.626 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:44:54.638 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":833,"stamp":"2024-09-16T11:44:54.437337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393496,"kb_used_data":278645352,"kb_used_omap":33825,"kb_used_meta":0,"kb_avail":278645352,"statfs":{"total":287783780352,"available":285332840448,"internally_reserved":0,"allocated":285332840448,"data_stored":285332840448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34637729,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002276"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.968952+0000","last_change":"2024-09-16T11:22:14.120245+0000","last_active":"2024-09-16T11:44:01.968952+0000","last_peered":"2024-09-16T11:44:01.968952+0000","last_clean":"2024-09-16T11:44:01.968952+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:44:01.968952+0000","last_undegraded":"2024-09-16T11:44:01.968952+0000","last_fullsized":"2024-09-16T11:44:01.968952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:50.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.971612+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:44:01.971612+0000","last_peered":"2024-09-16T11:44:01.971612+0000","last_clean":"2024-09-16T11:44:01.971612+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:44:01.971612+0000","last_undegraded":"2024-09-16T11:44:01.971612+0000","last_fullsized":"2024-09-16T11:44:01.971612+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:07.810548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.970998+0000","last_change":"2024-09-16T11:22:14.120010+0000","last_active":"2024-09-16T11:44:01.970998+0000","last_peered":"2024-09-16T11:44:01.970998+0000","last_clean":"2024-09-16T11:44:01.970998+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:44:01.970998+0000","last_undegraded":"2024-09-16T11:44:01.970998+0000","last_fullsized":"2024-09-16T11:44:01.970998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:43.889549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.969535+0000","last_change":"2024-09-16T11:22:14.119525+0000","last_active":"2024-09-16T11:44:01.969535+0000","last_peered":"2024-09-16T11:44:01.969535+0000","last_clean":"2024-09-16T11:44:01.969535+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:44:01.969535+0000","last_undegraded":"2024-09-16T11:44:01.969535+0000","last_fullsized":"2024-09-16T11:44:01.969535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:21.893309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":156,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.205119+0000","last_change":"2024-09-16T11:22:14.118460+0000","last_active":"2024-09-16T11:44:01.205119+0000","last_peered":"2024-09-16T11:44:01.205119+0000","last_clean":"2024-09-16T11:44:01.205119+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:44:01.205119+0000","last_undegraded":"2024-09-16T11:44:01.205119+0000","last_fullsized":"2024-09-16T11:44:01.205119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:29:26.098123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.548248+0000","last_change":"2024-09-16T11:22:14.116075+0000","last_active":"2024-09-16T11:44:01.548248+0000","last_peered":"2024-09-16T11:44:01.548248+0000","last_clean":"2024-09-16T11:44:01.548248+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:44:01.548248+0000","last_undegraded":"2024-09-16T11:44:01.548248+0000","last_fullsized":"2024-09-16T11:44:01.548248+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:59.034856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.550791+0000","last_change":"2024-09-16T11:22:14.116152+0000","last_active":"2024-09-16T11:44:01.550791+0000","last_peered":"2024-09-16T11:44:01.550791+0000","last_clean":"2024-09-16T11:44:01.550791+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:44:01.550791+0000","last_undegraded":"2024-09-16T11:44:01.550791+0000","last_fullsized":"2024-09-16T11:44:01.550791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:26.664156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":153,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.972784+0000","last_change":"2024-09-16T11:22:14.119568+0000","last_active":"2024-09-16T11:44:01.972784+0000","last_peered":"2024-09-16T11:44:01.972784+0000","last_clean":"2024-09-16T11:44:01.972784+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:44:01.972784+0000","last_undegraded":"2024-09-16T11:44:01.972784+0000","last_fullsized":"2024-09-16T11:44:01.972784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:46.653548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.973330+0000","last_change":"2024-09-16T11:22:10.960454+0000","last_active":"2024-09-16T11:44:01.973330+0000","last_peered":"2024-09-16T11:44:01.973330+0000","last_clean":"2024-09-16T11:44:01.973330+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:44:01.973330+0000","last_undegraded":"2024-09-16T11:44:01.973330+0000","last_fullsized":"2024-09-16T11:44:01.973330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_clean_scrub_stamp":"2024-09-16T11:22:09.097683+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:09.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738644,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797440,"kb_used_data":92882176,"kb_used_omap":10619,"kb_used_meta":0,"kb_avail":92882176,"statfs":{"total":95927926784,"available":95111348224,"internally_reserved":0,"allocated":95111348224,"data_stored":95111348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10874854,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.316,"15min":0.311},"min":{"1min":0.23100000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.33500000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.18099999999999999},"max":{"1min":0.47599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.317,"15min":0.32600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.51500000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738643,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":10885,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11146572,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.307},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.44700000000000001,"5min":0.497,"15min":0.497},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46400000000000002},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.19800000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.49199999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.189},"max":{"1min":0.441,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":12320,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12616303,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.30599999999999999},"min":{"1min":0.245,"5min":0.221,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.22,"15min":0.214},"max":{"1min":0.46000000000000002,"5min":0.48799999999999999,"15min":1.9279999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":0.436,"5min":1.026,"15min":1.026},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35299999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":0.495,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.25800000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:54.639 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T11:44:54.910 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:44:54.910 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":154,"fsid":"bfe848d9-9b22-4e38-923f-a2004deef198","created":"2024-09-16T11:22:02.047725+0000","modified":"2024-09-16T11:44:02.203477+0000","last_up_change":"2024-09-16T11:22:06.056486+0000","last_in_change":"2024-09-16T11:22:02.960807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T11:22:08.131565+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T11:22:10.731960+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9d8548df-278f-4b8b-a7c0-4faddbf572f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":28635},{"type":"v1","addr":"172.21.15.53:6809","nonce":28635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":28635},{"type":"v1","addr":"172.21.15.53:6811","nonce":28635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":28635},{"type":"v1","addr":"172.21.15.53:6815","nonce":28635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":28635},{"type":"v1","addr":"172.21.15.53:6813","nonce":28635}]},"public_addr":"172.21.15.53:6809/28635","cluster_addr":"172.21.15.53:6811/28635","heartbeat_back_addr":"172.21.15.53:6815/28635","heartbeat_front_addr":"172.21.15.53:6813/28635","state":["exists","up"]},{"osd":1,"uuid":"f9d9193d-5e46-49cc-ab26-34618db4a07d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":28626},{"type":"v1","addr":"172.21.15.53:6801","nonce":28626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":28626},{"type":"v1","addr":"172.21.15.53:6803","nonce":28626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":28626},{"type":"v1","addr":"172.21.15.53:6807","nonce":28626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":28626},{"type":"v1","addr":"172.21.15.53:6805","nonce":28626}]},"public_addr":"172.21.15.53:6801/28626","cluster_addr":"172.21.15.53:6803/28626","heartbeat_back_addr":"172.21.15.53:6807/28626","heartbeat_front_addr":"172.21.15.53:6805/28626","state":["exists","up"]},{"osd":2,"uuid":"635ae354-dc24-4826-9d97-3c5c80914da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":28638},{"type":"v1","addr":"172.21.15.53:6817","nonce":28638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":28638},{"type":"v1","addr":"172.21.15.53:6819","nonce":28638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":28638},{"type":"v1","addr":"172.21.15.53:6823","nonce":28638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":28638},{"type":"v1","addr":"172.21.15.53:6821","nonce":28638}]},"public_addr":"172.21.15.53:6817/28638","cluster_addr":"172.21.15.53:6819/28638","heartbeat_back_addr":"172.21.15.53:6823/28638","heartbeat_front_addr":"172.21.15.53:6821/28638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107284+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.107422+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T11:22:04.097413+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":4,"length":1},{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T11:44:55.924 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-16T11:44:55.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:44:56.056 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-16T11:44:56.056 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:44:56.056 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-16T11:44:56.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-16T11:44:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T11:44:56.352 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-16T11:44:56.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:44:56.471 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-16T11:44:56.471 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:44:56.471 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-16T11:44:56.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-16T11:44:56.755 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T11:44:56.768 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-16T11:44:56.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T11:44:56.887 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-09-16T11:44:56.887 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T11:44:56.887 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-09-16T11:44:56.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-16T11:44:57.170 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T11:44:57.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:44:57.450 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:44:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:44:57.462 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":834,"stamp":"2024-09-16T11:44:56.437655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393496,"kb_used_data":278645352,"kb_used_omap":33825,"kb_used_meta":0,"kb_avail":278645352,"statfs":{"total":287783780352,"available":285332840448,"internally_reserved":0,"allocated":285332840448,"data_stored":285332840448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34637729,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002234"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.968952+0000","last_change":"2024-09-16T11:22:14.120245+0000","last_active":"2024-09-16T11:44:01.968952+0000","last_peered":"2024-09-16T11:44:01.968952+0000","last_clean":"2024-09-16T11:44:01.968952+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:44:01.968952+0000","last_undegraded":"2024-09-16T11:44:01.968952+0000","last_fullsized":"2024-09-16T11:44:01.968952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:50.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.971612+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:44:01.971612+0000","last_peered":"2024-09-16T11:44:01.971612+0000","last_clean":"2024-09-16T11:44:01.971612+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:44:01.971612+0000","last_undegraded":"2024-09-16T11:44:01.971612+0000","last_fullsized":"2024-09-16T11:44:01.971612+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:07.810548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.970998+0000","last_change":"2024-09-16T11:22:14.120010+0000","last_active":"2024-09-16T11:44:01.970998+0000","last_peered":"2024-09-16T11:44:01.970998+0000","last_clean":"2024-09-16T11:44:01.970998+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:44:01.970998+0000","last_undegraded":"2024-09-16T11:44:01.970998+0000","last_fullsized":"2024-09-16T11:44:01.970998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:11:43.889549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.969535+0000","last_change":"2024-09-16T11:22:14.119525+0000","last_active":"2024-09-16T11:44:01.969535+0000","last_peered":"2024-09-16T11:44:01.969535+0000","last_clean":"2024-09-16T11:44:01.969535+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:44:01.969535+0000","last_undegraded":"2024-09-16T11:44:01.969535+0000","last_fullsized":"2024-09-16T11:44:01.969535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:21.893309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":156,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.205119+0000","last_change":"2024-09-16T11:22:14.118460+0000","last_active":"2024-09-16T11:44:01.205119+0000","last_peered":"2024-09-16T11:44:01.205119+0000","last_clean":"2024-09-16T11:44:01.205119+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:44:01.205119+0000","last_undegraded":"2024-09-16T11:44:01.205119+0000","last_fullsized":"2024-09-16T11:44:01.205119+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:29:26.098123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.548248+0000","last_change":"2024-09-16T11:22:14.116075+0000","last_active":"2024-09-16T11:44:01.548248+0000","last_peered":"2024-09-16T11:44:01.548248+0000","last_clean":"2024-09-16T11:44:01.548248+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:44:01.548248+0000","last_undegraded":"2024-09-16T11:44:01.548248+0000","last_fullsized":"2024-09-16T11:44:01.548248+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:59.034856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":152,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.550791+0000","last_change":"2024-09-16T11:22:14.116152+0000","last_active":"2024-09-16T11:44:01.550791+0000","last_peered":"2024-09-16T11:44:01.550791+0000","last_clean":"2024-09-16T11:44:01.550791+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:44:01.550791+0000","last_undegraded":"2024-09-16T11:44:01.550791+0000","last_fullsized":"2024-09-16T11:44:01.550791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:48:26.664156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":153,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.972784+0000","last_change":"2024-09-16T11:22:14.119568+0000","last_active":"2024-09-16T11:44:01.972784+0000","last_peered":"2024-09-16T11:44:01.972784+0000","last_clean":"2024-09-16T11:44:01.972784+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:44:01.972784+0000","last_undegraded":"2024-09-16T11:44:01.972784+0000","last_fullsized":"2024-09-16T11:44:01.972784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:46.653548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":216,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-16T11:44:01.973330+0000","last_change":"2024-09-16T11:22:10.960454+0000","last_active":"2024-09-16T11:44:01.973330+0000","last_peered":"2024-09-16T11:44:01.973330+0000","last_clean":"2024-09-16T11:44:01.973330+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:44:01.973330+0000","last_undegraded":"2024-09-16T11:44:01.973330+0000","last_fullsized":"2024-09-16T11:44:01.973330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:09.097683+0000","last_clean_scrub_stamp":"2024-09-16T11:22:09.097683+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:09.906077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738644,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797440,"kb_used_data":92882176,"kb_used_omap":10619,"kb_used_meta":0,"kb_avail":92882176,"statfs":{"total":95927926784,"available":95111348224,"internally_reserved":0,"allocated":95111348224,"data_stored":95111348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10874854,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.316,"15min":0.311},"min":{"1min":0.23100000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.33500000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.18099999999999999},"max":{"1min":0.47599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.317,"15min":0.32600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.51500000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738643,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":10885,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11146572,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.307},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.44700000000000001,"5min":0.497,"15min":0.497},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46400000000000002},"last":0.23699999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.19800000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.49199999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.189},"max":{"1min":0.441,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":12320,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12616303,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.30599999999999999},"min":{"1min":0.245,"5min":0.221,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.22,"15min":0.214},"max":{"1min":0.46000000000000002,"5min":0.48799999999999999,"15min":1.9279999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":0.436,"5min":1.026,"15min":1.026},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35299999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":0.495,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.25800000000000001}]}]}],"pool_statfs":[]}} 2024-09-16T11:44:57.463 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.463 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.463 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.463 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.463 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.464 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.464 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.464 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.464 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T11:22:09.097683+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=9, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:44:57.464 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:45:17.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:45:17.749 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:45:17.750 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:45:17.762 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":844,"stamp":"2024-09-16T11:45:16.441715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393508,"kb_used_data":278645352,"kb_used_omap":33853,"kb_used_meta":0,"kb_avail":278645340,"statfs":{"total":287783780352,"available":285332828160,"internally_reserved":12288,"allocated":285332840448,"data_stored":285332840448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34666384,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002461"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:06.585278+0000","last_change":"2024-09-16T11:45:06.585253+0000","last_active":"2024-09-16T11:45:06.585278+0000","last_peered":"2024-09-16T11:45:06.585278+0000","last_clean":"2024-09-16T11:45:06.585278+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:45:06.585278+0000","last_undegraded":"2024-09-16T11:45:06.585278+0000","last_fullsized":"2024-09-16T11:45:06.585278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:45:06.585186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:45:06.585186+0000","last_clean_scrub_stamp":"2024-09-16T11:45:06.585186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:21.697914+0000","scrub_duration":0.0015456070000000001,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":153,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:02.641367+0000","last_change":"2024-09-16T11:22:14.119916+0000","last_active":"2024-09-16T11:44:02.641367+0000","last_peered":"2024-09-16T11:44:02.641367+0000","last_clean":"2024-09-16T11:44:02.641367+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:44:02.641367+0000","last_undegraded":"2024-09-16T11:44:02.641367+0000","last_fullsized":"2024-09-16T11:44:02.641367+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:22:11.102510+0000","last_clean_scrub_stamp":"2024-09-16T11:22:11.102510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:08.569771+0000","last_change":"2024-09-16T11:45:08.569743+0000","last_active":"2024-09-16T11:45:08.569771+0000","last_peered":"2024-09-16T11:45:08.569771+0000","last_clean":"2024-09-16T11:45:08.569771+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:45:08.569771+0000","last_undegraded":"2024-09-16T11:45:08.569771+0000","last_fullsized":"2024-09-16T11:45:08.569771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:45:08.569679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:45:08.569679+0000","last_clean_scrub_stamp":"2024-09-16T11:45:08.569679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:07:47.751830+0000","scrub_duration":0.001470364,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:59.621021+0000","last_change":"2024-09-16T11:44:59.620994+0000","last_active":"2024-09-16T11:44:59.621021+0000","last_peered":"2024-09-16T11:44:59.621021+0000","last_clean":"2024-09-16T11:44:59.621021+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:44:59.621021+0000","last_undegraded":"2024-09-16T11:44:59.621021+0000","last_fullsized":"2024-09-16T11:44:59.621021+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:59.620934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:59.620934+0000","last_clean_scrub_stamp":"2024-09-16T11:44:59.620934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:22:55.289436+0000","scrub_duration":0.0013677610000000001,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"154'3","reported_seq":168,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:59.544504+0000","last_change":"2024-09-16T11:44:59.544479+0000","last_active":"2024-09-16T11:44:59.544504+0000","last_peered":"2024-09-16T11:44:59.544504+0000","last_clean":"2024-09-16T11:44:59.544504+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:44:59.544504+0000","last_undegraded":"2024-09-16T11:44:59.544504+0000","last_fullsized":"2024-09-16T11:44:59.544504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'3","last_scrub_stamp":"2024-09-16T11:44:59.544416+0000","last_deep_scrub":"154'3","last_deep_scrub_stamp":"2024-09-16T11:44:59.544416+0000","last_clean_scrub_stamp":"2024-09-16T11:44:59.544416+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:52.905256+0000","scrub_duration":0.0024438329999999999,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:59.271368+0000","last_change":"2024-09-16T11:44:59.271324+0000","last_active":"2024-09-16T11:44:59.271368+0000","last_peered":"2024-09-16T11:44:59.271368+0000","last_clean":"2024-09-16T11:44:59.271368+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:44:59.271368+0000","last_undegraded":"2024-09-16T11:44:59.271368+0000","last_fullsized":"2024-09-16T11:44:59.271368+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:59.271260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:59.271260+0000","last_clean_scrub_stamp":"2024-09-16T11:44:59.271260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:23:51.317311+0000","scrub_duration":0.0015296730000000001,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:58.254921+0000","last_change":"2024-09-16T11:44:58.254895+0000","last_active":"2024-09-16T11:44:58.254921+0000","last_peered":"2024-09-16T11:44:58.254921+0000","last_clean":"2024-09-16T11:44:58.254921+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:44:58.254921+0000","last_undegraded":"2024-09-16T11:44:58.254921+0000","last_fullsized":"2024-09-16T11:44:58.254921+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:58.254832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:58.254832+0000","last_clean_scrub_stamp":"2024-09-16T11:44:58.254832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:23.306708+0000","scrub_duration":0.001493528,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"154'2","reported_seq":165,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:00.661436+0000","last_change":"2024-09-16T11:45:00.661385+0000","last_active":"2024-09-16T11:45:00.661436+0000","last_peered":"2024-09-16T11:45:00.661436+0000","last_clean":"2024-09-16T11:45:00.661436+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:45:00.661436+0000","last_undegraded":"2024-09-16T11:45:00.661436+0000","last_fullsized":"2024-09-16T11:45:00.661436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-16T11:45:00.661320+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-16T11:45:00.661320+0000","last_clean_scrub_stamp":"2024-09-16T11:45:00.661320+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:31.354817+0000","scrub_duration":0.0025063910000000002,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"154'54","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:58.636460+0000","last_change":"2024-09-16T11:44:58.636431+0000","last_active":"2024-09-16T11:44:58.636460+0000","last_peered":"2024-09-16T11:44:58.636460+0000","last_clean":"2024-09-16T11:44:58.636460+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:44:58.636460+0000","last_undegraded":"2024-09-16T11:44:58.636460+0000","last_fullsized":"2024-09-16T11:44:58.636460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'54","last_scrub_stamp":"2024-09-16T11:44:58.636367+0000","last_deep_scrub":"154'54","last_deep_scrub_stamp":"2024-09-16T11:44:58.636367+0000","last_clean_scrub_stamp":"2024-09-16T11:44:58.636367+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:52.749672+0000","scrub_duration":0.0037341919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738648,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797440,"kb_used_data":92882176,"kb_used_omap":10624,"kb_used_meta":0,"kb_avail":92882176,"statfs":{"total":95927926784,"available":95111348224,"internally_reserved":0,"allocated":95111348224,"data_stored":95111348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10879854,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.316,"15min":0.311},"min":{"1min":0.23100000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.33500000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.18099999999999999},"max":{"1min":0.47599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.317,"15min":0.32600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.51500000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.29599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738647,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798036,"kb_used_data":92881588,"kb_used_omap":10899,"kb_used_meta":0,"kb_avail":92881580,"statfs":{"total":95927926784,"available":95110737920,"internally_reserved":8192,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11160795,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.307},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.44700000000000001,"5min":0.497,"15min":0.497},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46400000000000002},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.19800000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.49199999999999999},"last":0.315},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.189},"max":{"1min":0.441,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738647,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798032,"kb_used_data":92881588,"kb_used_omap":12329,"kb_used_meta":0,"kb_avail":92881584,"statfs":{"total":95927926784,"available":95110742016,"internally_reserved":4096,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12625735,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.30599999999999999},"min":{"1min":0.245,"5min":0.221,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.22,"15min":0.214},"max":{"1min":0.46000000000000002,"5min":0.48799999999999999,"15min":1.9279999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":0.436,"5min":1.026,"15min":1.026},"last":0.253},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35299999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":0.495,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.27000000000000002}]}]}],"pool_statfs":[]}} 2024-09-16T11:45:17.763 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T11:22:11.102510+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=22, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=11, tm_min=44, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T11:45:17.763 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T11:45:37.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T11:45:38.051 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-09-16T11:45:38.052 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-09-16T11:45:38.063 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":854,"stamp":"2024-09-16T11:45:36.445595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281038848,"kb_used":2393496,"kb_used_data":278645352,"kb_used_omap":33856,"kb_used_meta":0,"kb_avail":278645352,"statfs":{"total":287783780352,"available":285332840448,"internally_reserved":0,"allocated":285332840448,"data_stored":285332840448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34668991,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002248"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:06.585278+0000","last_change":"2024-09-16T11:45:06.585253+0000","last_active":"2024-09-16T11:45:06.585278+0000","last_peered":"2024-09-16T11:45:06.585278+0000","last_clean":"2024-09-16T11:45:06.585278+0000","last_became_active":"2024-09-16T11:22:12.115541+0000","last_became_peered":"2024-09-16T11:22:12.115541+0000","last_unstale":"2024-09-16T11:45:06.585278+0000","last_undegraded":"2024-09-16T11:45:06.585278+0000","last_fullsized":"2024-09-16T11:45:06.585278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:45:06.585186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:45:06.585186+0000","last_clean_scrub_stamp":"2024-09-16T11:45:06.585186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:21.697914+0000","scrub_duration":0.0015456070000000001,"objects_trimmed":0,"snaptrim_duration":0.001205058,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:13.554907+0000","last_change":"2024-09-16T11:45:13.554879+0000","last_active":"2024-09-16T11:45:13.554907+0000","last_peered":"2024-09-16T11:45:13.554907+0000","last_clean":"2024-09-16T11:45:13.554907+0000","last_became_active":"2024-09-16T11:22:12.114555+0000","last_became_peered":"2024-09-16T11:22:12.114555+0000","last_unstale":"2024-09-16T11:45:13.554907+0000","last_undegraded":"2024-09-16T11:45:13.554907+0000","last_fullsized":"2024-09-16T11:45:13.554907+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:45:13.554814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:45:13.554814+0000","last_clean_scrub_stamp":"2024-09-16T11:45:13.554814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:43:44.969845+0000","scrub_duration":0.001529795,"objects_trimmed":0,"snaptrim_duration":0.00080996300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:08.569771+0000","last_change":"2024-09-16T11:45:08.569743+0000","last_active":"2024-09-16T11:45:08.569771+0000","last_peered":"2024-09-16T11:45:08.569771+0000","last_clean":"2024-09-16T11:45:08.569771+0000","last_became_active":"2024-09-16T11:22:12.116199+0000","last_became_peered":"2024-09-16T11:22:12.116199+0000","last_unstale":"2024-09-16T11:45:08.569771+0000","last_undegraded":"2024-09-16T11:45:08.569771+0000","last_fullsized":"2024-09-16T11:45:08.569771+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:45:08.569679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:45:08.569679+0000","last_clean_scrub_stamp":"2024-09-16T11:45:08.569679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:07:47.751830+0000","scrub_duration":0.001470364,"objects_trimmed":0,"snaptrim_duration":0.00088780499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:59.621021+0000","last_change":"2024-09-16T11:44:59.620994+0000","last_active":"2024-09-16T11:44:59.621021+0000","last_peered":"2024-09-16T11:44:59.621021+0000","last_clean":"2024-09-16T11:44:59.621021+0000","last_became_active":"2024-09-16T11:22:12.117454+0000","last_became_peered":"2024-09-16T11:22:12.117454+0000","last_unstale":"2024-09-16T11:44:59.621021+0000","last_undegraded":"2024-09-16T11:44:59.621021+0000","last_fullsized":"2024-09-16T11:44:59.621021+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:59.620934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:59.620934+0000","last_clean_scrub_stamp":"2024-09-16T11:44:59.620934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:22:55.289436+0000","scrub_duration":0.0013677610000000001,"objects_trimmed":0,"snaptrim_duration":0.00055023799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"154'3","reported_seq":168,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:59.544504+0000","last_change":"2024-09-16T11:44:59.544479+0000","last_active":"2024-09-16T11:44:59.544504+0000","last_peered":"2024-09-16T11:44:59.544504+0000","last_clean":"2024-09-16T11:44:59.544504+0000","last_became_active":"2024-09-16T11:22:12.117014+0000","last_became_peered":"2024-09-16T11:22:12.117014+0000","last_unstale":"2024-09-16T11:44:59.544504+0000","last_undegraded":"2024-09-16T11:44:59.544504+0000","last_fullsized":"2024-09-16T11:44:59.544504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'3","last_scrub_stamp":"2024-09-16T11:44:59.544416+0000","last_deep_scrub":"154'3","last_deep_scrub_stamp":"2024-09-16T11:44:59.544416+0000","last_clean_scrub_stamp":"2024-09-16T11:44:59.544416+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:47:52.905256+0000","scrub_duration":0.0024438329999999999,"objects_trimmed":0,"snaptrim_duration":0.00070361499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:59.271368+0000","last_change":"2024-09-16T11:44:59.271324+0000","last_active":"2024-09-16T11:44:59.271368+0000","last_peered":"2024-09-16T11:44:59.271368+0000","last_clean":"2024-09-16T11:44:59.271368+0000","last_became_active":"2024-09-16T11:22:12.115221+0000","last_became_peered":"2024-09-16T11:22:12.115221+0000","last_unstale":"2024-09-16T11:44:59.271368+0000","last_undegraded":"2024-09-16T11:44:59.271368+0000","last_fullsized":"2024-09-16T11:44:59.271368+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:59.271260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:59.271260+0000","last_clean_scrub_stamp":"2024-09-16T11:44:59.271260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:23:51.317311+0000","scrub_duration":0.0015296730000000001,"objects_trimmed":0,"snaptrim_duration":0.00036984799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":163,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:58.254921+0000","last_change":"2024-09-16T11:44:58.254895+0000","last_active":"2024-09-16T11:44:58.254921+0000","last_peered":"2024-09-16T11:44:58.254921+0000","last_clean":"2024-09-16T11:44:58.254921+0000","last_became_active":"2024-09-16T11:22:12.114854+0000","last_became_peered":"2024-09-16T11:22:12.114854+0000","last_unstale":"2024-09-16T11:44:58.254921+0000","last_undegraded":"2024-09-16T11:44:58.254921+0000","last_fullsized":"2024-09-16T11:44:58.254921+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T11:44:58.254832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T11:44:58.254832+0000","last_clean_scrub_stamp":"2024-09-16T11:44:58.254832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:54:23.306708+0000","scrub_duration":0.001493528,"objects_trimmed":0,"snaptrim_duration":0.00043460900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"154'2","reported_seq":165,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:45:00.661436+0000","last_change":"2024-09-16T11:45:00.661385+0000","last_active":"2024-09-16T11:45:00.661436+0000","last_peered":"2024-09-16T11:45:00.661436+0000","last_clean":"2024-09-16T11:45:00.661436+0000","last_became_active":"2024-09-16T11:22:12.113746+0000","last_became_peered":"2024-09-16T11:22:12.113746+0000","last_unstale":"2024-09-16T11:45:00.661436+0000","last_undegraded":"2024-09-16T11:45:00.661436+0000","last_fullsized":"2024-09-16T11:45:00.661436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-16T11:45:00.661320+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-16T11:45:00.661320+0000","last_clean_scrub_stamp":"2024-09-16T11:45:00.661320+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:42:31.354817+0000","scrub_duration":0.0025063910000000002,"objects_trimmed":0,"snaptrim_duration":0.00061449900000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"154'54","reported_seq":229,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T11:44:58.636460+0000","last_change":"2024-09-16T11:44:58.636431+0000","last_active":"2024-09-16T11:44:58.636460+0000","last_peered":"2024-09-16T11:44:58.636460+0000","last_clean":"2024-09-16T11:44:58.636460+0000","last_became_active":"2024-09-16T11:22:10.959696+0000","last_became_peered":"2024-09-16T11:22:10.959696+0000","last_unstale":"2024-09-16T11:44:58.636460+0000","last_undegraded":"2024-09-16T11:44:58.636460+0000","last_fullsized":"2024-09-16T11:44:58.636460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'54","last_scrub_stamp":"2024-09-16T11:44:58.636367+0000","last_deep_scrub":"154'54","last_deep_scrub_stamp":"2024-09-16T11:44:58.636367+0000","last_clean_scrub_stamp":"2024-09-16T11:44:58.636367+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:32:52.749672+0000","scrub_duration":0.0037341919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738652,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797440,"kb_used_data":92882176,"kb_used_omap":10624,"kb_used_meta":0,"kb_avail":92882176,"statfs":{"total":95927926784,"available":95111348224,"internally_reserved":0,"allocated":95111348224,"data_stored":95111348224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10879854,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.316,"15min":0.311},"min":{"1min":0.23100000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.47399999999999998,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.33500000000000002},"min":{"1min":0.25700000000000001,"5min":0.218,"15min":0.18099999999999999},"max":{"1min":0.47599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.27300000000000002}]},{"osd":1,"last update":"Mon Sep 16 11:44:47 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.317,"15min":0.32600000000000001},"min":{"1min":0.23300000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.51500000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738651,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":10900,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162062,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.30599999999999999,"15min":0.307},"min":{"1min":0.183,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":0.44700000000000001,"5min":0.497,"15min":0.497},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.20799999999999999,"15min":0.20300000000000001},"max":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46400000000000002},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Sep 16 11:44:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.33200000000000002},"min":{"1min":0.19800000000000001,"5min":0.183,"15min":0.183},"max":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.49199999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35299999999999998,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.189},"max":{"1min":0.441,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738651,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":798028,"kb_used_data":92881588,"kb_used_omap":12331,"kb_used_meta":0,"kb_avail":92881588,"statfs":{"total":95927926784,"available":95110746112,"internally_reserved":0,"allocated":95110746112,"data_stored":95110746112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12627075,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.31900000000000001,"15min":0.30599999999999999},"min":{"1min":0.245,"5min":0.221,"15min":0.187},"max":{"1min":0.47199999999999998,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33100000000000002,"15min":0.33200000000000002},"min":{"1min":0.223,"5min":0.22,"15min":0.214},"max":{"1min":0.46000000000000002,"5min":0.48799999999999999,"15min":1.9279999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Sep 16 11:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.32200000000000001},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20599999999999999},"max":{"1min":0.436,"5min":1.026,"15min":1.026},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35299999999999998},"min":{"1min":0.29099999999999998,"5min":0.25700000000000001,"15min":0.222},"max":{"1min":0.495,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.46999999999999997}]}]}],"pool_statfs":[]}} 2024-09-16T11:45:38.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-16T11:45:38.348 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T11:45:38.349 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T11:45:38.349 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-16T11:45:38.349 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:45:38.364 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-16T11:45:38.365 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-16T11:45:38.365 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:45:38.420 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-16T11:45:38.421 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-16T11:45:38.421 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:45:38.475 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-16T11:45:38.475 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T11:45:38.475 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-16T11:45:38.475 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:45:38.539 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-16T11:45:38.540 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T11:45:38.540 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-16T11:45:38.540 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T11:45:38.591 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-16T11:45:38.591 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T11:45:38.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T11:45:38.659 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:45:38.659 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-16T11:45:38.790 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:45:38.790 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-16T11:45:38.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:45:38.864 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-16T11:45:38.929 INFO:tasks.ceph:Archiving mon data... 2024-09-16T11:45:38.930 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/data/cluster1.mon.a.tgz 2024-09-16T11:45:38.930 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-09-16T11:45:38.946 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.FWLf5YZe2r 2024-09-16T11:45:38.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.FWLf5YZe2r 2024-09-16T11:45:39.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.FWLf5YZe2r 2024-09-16T11:45:39.176 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.FWLf5YZe2r is 357KB 2024-09-16T11:45:39.243 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.FWLf5YZe2r 2024-09-16T11:45:39.259 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T11:45:39.259 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:45:39.301 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T11:45:39.333 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T11:45:39.333 DEBUG:teuthology.orchestra.run.smithi053:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T11:45:39.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T11:45:39.391 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T11:45:39.405 INFO:teuthology.orchestra.run.smithi204.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T11:45:39.410 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T11:45:39.412 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/remote/smithi053/crash 2024-09-16T11:45:39.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T11:45:39.462 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/remote/smithi204/crash 2024-09-16T11:45:39.463 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T11:45:39.490 INFO:tasks.ceph:Compressing logs... 2024-09-16T11:45:39.491 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T11:45:39.507 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T11:45:39.529 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30762.log 2024-09-16T11:45:39.529 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29298.log 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29256.log 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30762.log.gz 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29548.log 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29298.log.gz 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30804.log 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29256.log.gz 2024-09-16T11:45:39.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30426.log 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29548.log.gz 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30594.log 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30804.log.gz 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30426.log.gz 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29655.log 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30594.log.gz 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29422.log 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29655.log.gz 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29422.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29422.log.gz 2024-09-16T11:45:39.549 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29844.log 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28949.log 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28288.log 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29844.log.gz 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.28949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30510.log 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28949.log.gz 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.28288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28526.log 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28288.log.gz 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30510.log.gz 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27001.log 2024-09-16T11:45:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.28526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30844.log 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28526.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.tmp-client.admin.27001.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.27001.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.29547.log 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30342.log 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30844.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29547.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30468.log 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29421.log 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30342.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29136.log 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30468.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29421.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29421.log.gz 2024-09-16T11:45:39.551 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30300.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29136.log.gz 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30300.log.gz 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30552.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29340.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30678.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30552.log.gz 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29420.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29340.log.gz 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30678.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30678.log.gz 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.30384.log 2024-09-16T11:45:39.552 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29420.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29420.log.gz 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30384.log.gz 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30720.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29508.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30720.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30720.log.gz 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28625.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29508.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29508.log.gz 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr: -- /var/log/ceph/cluster1-client.admin.29182.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.28625.log: 28.5% -- replaced with /var/log/ceph/cluster1-client.admin.28625.log.gz 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28486.log 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.29182.log.gz 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29634.log 2024-09-16T11:45:39.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.28486.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29802.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.28486.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29634.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29802.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30258.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.29802.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28446.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30258.log: /var/log/ceph/cluster1-osd.0.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30258.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29760.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.28446.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28446.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29694.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29760.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30636.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29694.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29694.log.gz 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-16T11:45:39.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.30636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30636.log.gz 2024-09-16T11:45:39.555 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29078.log 2024-09-16T11:45:39.556 INFO:teuthology.orchestra.run.smithi053.stderr: 92.5% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-16T11:45:39.556 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1.log: 87.5% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-16T11:45:39.556 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cluster1-client.admin.29078.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29078.log.gz 2024-09-16T11:45:39.562 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67411.log 2024-09-16T11:45:39.563 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60167.log 2024-09-16T11:45:39.564 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.67411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59051.log 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40389.log 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60167.log: /var/log/ceph/cluster2-client.mirror.59051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66339.log 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60167.log.gz 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.40389.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59051.log.gz 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57768.log 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67411.log.gz 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40389.log.gz 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41801.log 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66339.log.gz 2024-09-16T11:45:39.565 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53176.log 2024-09-16T11:45:39.572 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.41801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64194.log 2024-09-16T11:45:39.572 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.57768.log.gz 2024-09-16T11:45:39.572 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41801.log.gz 2024-09-16T11:45:39.572 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.53176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60217.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.64194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31695.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53176.log.gz 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60217.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64194.log.gz 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72947.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.31695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36070.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31695.log.gz 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60217.log.gz 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.mirror.72947.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32339.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72947.log.gz 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.0.36070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51844.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70840.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.51844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60243.log 2024-09-16T11:45:39.573 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70840.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51844.log.gz 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49802.log 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70840.log.gz 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51500.log 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60243.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.32339.log.gz 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.49802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33455.log 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60243.log.gz 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51500.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49802.log.gz 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47841.log 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33455.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.51500.log.gz 2024-09-16T11:45:39.574 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58610.log 2024-09-16T11:45:39.679 INFO:teuthology.orchestra.run.smithi053.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-16T11:45:39.736 INFO:teuthology.orchestra.run.smithi053.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-16T11:45:39.976 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73768.log 2024-09-16T11:45:39.976 INFO:teuthology.orchestra.run.smithi204.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.33455.log.gz 2024-09-16T11:45:39.976 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.58610.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.47841.log.gz 2024-09-16T11:45:39.976 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37003.log 2024-09-16T11:45:39.976 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73768.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73768.log.gz 2024-09-16T11:45:39.977 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73120.log 2024-09-16T11:45:39.977 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.admin.58610.log.gz 2024-09-16T11:45:39.977 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58204.log 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73120.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.37003.log.gz 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31637.log 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73120.log.gz 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61816.log 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58204.log.gz 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31849.log 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31637.log.gz 2024-09-16T11:45:39.980 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33593.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61816.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42364.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38630.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31849.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33593.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.42364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59439.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38630.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.42364.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72490.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38630.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45559.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54025.log 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59439.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.admin.72490.log.gz 2024-09-16T11:45:39.981 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30548.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.45559.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72312.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30548.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.54025.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.30548.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.68704.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42817.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.68704.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72312.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63813.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42817.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.68704.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57797.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.63813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30042.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42817.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.63813.log.gz 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.57797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49527.log 2024-09-16T11:45:39.982 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53569.log 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.57797.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30042.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.49527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70757.log 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53569.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49527.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60726.log 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70757.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.53569.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56793.log 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70757.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58662.log 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60726.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57427.log 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58662.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.56793.log.gz 2024-09-16T11:45:39.983 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54661.log 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.57427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69814.log 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.54661.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.57427.log.gz 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39986.log 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54661.log.gz 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57250.log 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69814.log: /var/log/ceph/cluster2-client.mirror.39986.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69814.log.gz 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46616.log 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39986.log.gz 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31918.log 2024-09-16T11:45:39.984 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.57250.log: /var/log/ceph/cluster1-client.admin.46616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60384.log 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.57250.log.gz 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31918.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.46616.log.gz 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53965.log 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70875.log 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31918.log.gz 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56824.log 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60384.log.gz 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70875.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.53965.log.gz 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42133.log 2024-09-16T11:45:39.985 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70875.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34276.log 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56824.log: /var/log/ceph/cluster1-client.mirror.42133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50715.log 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.56824.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71947.log 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42133.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.50715.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.34276.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32801.log 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72047.log 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71947.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50715.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42542.log 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71947.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32801.log.gz 2024-09-16T11:45:39.986 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59216.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72047.log: /var/log/ceph/cluster2-client.admin.42542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45329.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72047.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34646.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.42542.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48309.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58662.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41226.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.45329.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48309.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34646.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.59216.log.gz -5 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.48280.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.48309.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.41226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68108.log 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48280.log: 88.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.41226.log.gz 2024-09-16T11:45:39.987 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55064.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69406.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.48280.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.55064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63008.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55064.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72132.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63008.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69406.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34183.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63008.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68108.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40518.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72132.log: /var/log/ceph/cluster1-client.admin.34183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68861.log 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72132.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40518.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.34183.log.gz 2024-09-16T11:45:39.988 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51938.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.40518.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30002.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68861.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.51938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52616.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30002.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51938.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41332.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30002.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65918.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.41332.log: 87.7%gzip -- replaced with /var/log/ceph/cluster1-client.admin.52616.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.72655.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.65918.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41332.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44741.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65918.log.gz 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.0.72655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57090.log 2024-09-16T11:45:39.989 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.72655.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.44741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33857.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.57090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31415.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57090.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33857.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44741.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52969.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63493.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.31415.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33857.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31415.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51621.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50627.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.52969.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.51621.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63493.log.gz 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42239.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63271.log 2024-09-16T11:45:39.990 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.42239.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.50627.log.gz 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51621.log.gz 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44713.log 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42239.log.gz 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59491.log 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63271.log.gz 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.44713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36431.log 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65874.log 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.44713.log.gz 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.36431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58847.log 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65874.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59491.log.gz 2024-09-16T11:45:39.991 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66875.log 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.36431.log.gz 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41766.log 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65874.log.gz 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69839.log 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.41766.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66875.log.gz 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58847.log.gz 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51593.log 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41766.log.gz 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72344.log 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51593.log: gzip -5 --verbose -- 88.0% /var/log/ceph/cluster2-client.admin.52710.log -- replaced with /var/log/ceph/cluster1-client.mirror.69839.log.gz 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.32553.log 2024-09-16T11:45:39.992 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.51593.log.gz 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52710.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72344.log.gz 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34815.log 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.52710.log.gz 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.32553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73022.log 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59416.log 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34815.log.gz 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.73022.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.73022.log.gz 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59949.log 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66734.log 2024-09-16T11:45:39.993 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38567.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30188.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66734.log.gz 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59949.log.gz 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.38567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39641.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.38567.log.gz 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61458.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30188.log.gz 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59416.log.gz 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43805.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39641.log: /var/log/ceph/cluster2-client.mirror.61458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43924.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.39641.log.gz 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54814.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44150.log 2024-09-16T11:45:39.994 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43805.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.54814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59130.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.43924.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.44150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61887.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.59130.log: 89.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44346.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54814.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.44150.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35484.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.59130.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.44346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50217.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.35484.log: gzip -5 0.0% --verbose -- /var/log/ceph/cluster2-client.mirror.39231.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44346.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35484.log.gz 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60264.log 2024-09-16T11:45:39.995 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.50217.log.gz 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.39231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58546.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44401.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.58546.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39231.log.gz 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43226.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60264.log.gz 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62852.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48050.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.58546.log.gz 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43226.log: /var/log/ceph/cluster1-client.mirror.62852.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44401.log.gz 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65478.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7%/var/log/ceph/cluster2-client.admin.48050.log: -- replaced with /var/log/ceph/cluster1-client.mirror.62852.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32833.log 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43226.log.gz 2024-09-16T11:45:39.996 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.65478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68655.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.48050.log.gz 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.32833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69383.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46528.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.32833.log.gz 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65478.log.gz 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68655.log: /var/log/ceph/cluster1-client.mirror.69383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68463.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61458.log.gz 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68655.log.gz 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.46528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54874.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.68463.log: --verbose 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.46528.log.gz 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.mirror.45026.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61141.log 2024-09-16T11:45:39.997 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69383.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.54874.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.45026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55625.log 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45026.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54508.log 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61887.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68463.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61141.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.55625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59086.log 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55625.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55813.log 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66844.log 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54508.log.gz 2024-09-16T11:45:39.998 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59086.log.gz 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.55813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47812.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30464.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66844.log: /var/log/ceph/cluster1-client.admin.47812.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55813.log.gz 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66844.log.gz 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32359.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.47812.log.gz 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58577.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.30464.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30464.log.gz 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.32359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40681.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.58577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64229.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.58577.log.gz 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38365.log 2024-09-16T11:45:39.999 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69426.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.64229.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.40681.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38365.log: gzip -5 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64229.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.63864.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32359.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64261.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29750.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63864.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.64261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56144.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38365.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.64261.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63209.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29750.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29750.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.56144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30380.log 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56144.log.gz 2024-09-16T11:45:40.000 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59396.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69426.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63209.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.30380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38473.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30380.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34088.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.38473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29834.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.38473.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64586.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37891.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29834.log: 88.0% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34088.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.29834.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.64586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33624.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64586.log.gz 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40111.log 2024-09-16T11:45:40.001 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59396.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33624.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.37891.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37498.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64418.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40111.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.33624.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.37498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41426.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.64418.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40111.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37498.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40080.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41426.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.64418.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.32530.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.41426.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72104.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.31732.log 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.32530.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40080.log.gz 2024-09-16T11:45:40.002 INFO:teuthology.orchestra.run.smithi204.stderr: 18.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.32530.log.gz 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41695.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.66415.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.72104.log.gz 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.31732.log: /var/log/ceph/cluster2-client.mirror.41695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50188.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.66415.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41695.log.gz 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61993.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41667.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50188.log: /var/log/ceph/cluster2-client.mirror.61993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53148.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.50188.log.gz 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40290.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53148.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.41667.log.gz 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48548.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9%/var/log/ceph/cluster2-client.admin.40290.log: -- replaced with /var/log/ceph/cluster2-client.admin.53148.log.gz 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47341.log 2024-09-16T11:45:40.003 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.40290.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66649.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.48548.log: /var/log/ceph/cluster2-client.admin.47341.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48548.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.47341.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61261.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44861.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2%/var/log/ceph/cluster1-client.mirror.61261.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66649.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34245.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72699.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44861.log: /var/log/ceph/cluster1-client.admin.34245.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.44861.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.34245.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61261.log.gz 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72542.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70388.log 2024-09-16T11:45:40.004 INFO:teuthology.orchestra.run.smithi204.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72699.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68596.log 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.70388.log: -- replaced with /var/log/ceph/cluster1-client.admin.72542.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39351.log 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70388.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54626.log 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68596.log: /var/log/ceph/cluster2-client.admin.39351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43645.log 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7%/var/log/ceph/cluster2-client.mirror.54626.log: -- replaced with /var/log/ceph/cluster1-client.mirror.68596.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61993.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.39351.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54626.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65808.log 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.43645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62874.log 87.4% 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43645.log.gz 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62807.log 2024-09-16T11:45:40.005 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62938.log 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62874.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3%/var/log/ceph/cluster1-client.mirror.62807.log: -- replaced with /var/log/ceph/cluster2-client.mirror.65808.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70972.log 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62807.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49156.log 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62938.log: /var/log/ceph/cluster2-client.mirror.70972.log: 92.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67198.log 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.31732.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70972.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34537.log 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62938.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57031.log 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.49156.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67198.log.gz 2024-09-16T11:45:40.006 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47081.log 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.57031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50656.log 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.34537.log.gz/var/log/ceph/cluster1-client.admin.47081.log: 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65830.log 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57031.log.gz 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.admin.50656.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58917.log 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.47081.log.gz 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47144.log 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.50656.log.gz 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70419.log 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65830.log.gz 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58917.log.gz 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54755.log 2024-09-16T11:45:40.007 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71207.log 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.47144.log.gz 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49656.log 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70419.log.gz 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60428.log 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.54755.log.gz 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49656.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71207.log.gz 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46266.log 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.49656.log.gz 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55785.log 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62016.log 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60428.log.gz 2024-09-16T11:45:40.008 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46266.log: /var/log/ceph/cluster1-client.admin.55785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37831.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46266.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.55785.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42848.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62016.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71088.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37831.log: /var/log/ceph/cluster1-client.admin.42848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57218.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.37831.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42848.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72516.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31231.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71088.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.57218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50123.log 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72516.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57218.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.31231.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31231.log.gz 2024-09-16T11:45:40.009 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.37063.log 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72516.log.gz 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.50123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63953.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50123.log.gz 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51433.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.37063.log.gz 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64453.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.51433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29915.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51433.log.gz 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.64453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40707.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29915.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29915.log.gz 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45120.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31255.log 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45120.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.40707.log.gz 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.64453.log.gz 2024-09-16T11:45:40.010 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31296.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.45120.log.gz 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53905.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.31255.log: /var/log/ceph/cluster1-client.admin.31296.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31255.log.gz 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48429.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.31296.log.gz 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.53905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35712.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64292.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48081.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6% 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.48429.log.gz 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53905.log.gz 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.64292.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35406.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48851.log 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.35406.log: -- replaced with /var/log/ceph/cluster1-client.admin.64292.log.gz 2024-09-16T11:45:40.011 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.48081.log.gz 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35406.log.gz 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71733.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41542.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71733.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.48851.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69931.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71733.log.gz 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57880.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41542.log: /var/log/ceph/cluster1-client.mirror.69931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70625.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6%/var/log/ceph/cluster2-client.mirror.57880.log: -- replaced with /var/log/ceph/cluster1-client.mirror.69931.log.gz 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41542.log.gz 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66928.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70625.log: 94.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73096.log 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63953.log.gz 2024-09-16T11:45:40.012 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70625.log.gz 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55873.log 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr: 84.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66928.log.gz/var/log/ceph/cluster1-client.admin.73096.log: 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73096.log.gz 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52211.log 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46707.log 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52211.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.55873.log.gz 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35745.log 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46707.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.52211.log.gz 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59768.log 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42601.log 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.46707.log.gz 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35745.log: /var/log/ceph/cluster2-client.mirror.59768.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35745.log.gz 2024-09-16T11:45:40.013 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35712.log.gz -5 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.71450.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59768.log.gz 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.mirror.42601.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55567.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71450.log: gzip 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42601.log.gz 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49924.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71450.log.gz 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.55567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60470.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.49924.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55567.log.gz 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.32516.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60470.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49924.log.gz 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41602.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.32516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29046.log 2024-09-16T11:45:40.014 INFO:teuthology.orchestra.run.smithi204.stderr: 18.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.32516.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57371.log 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54478.log 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0%/var/log/ceph/cluster2-client.admin.29046.log: -- replaced with /var/log/ceph/cluster2-client.admin.41602.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.57371.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29046.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60470.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr: /var/log/ceph/cluster2-client.mirror.67121.log 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57880.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35780.log 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57371.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54478.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66775.log 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35780.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67121.log.gz 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39150.log 2024-09-16T11:45:40.015 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35780.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43984.log 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66775.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61909.log 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38166.log 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.39150.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45266.log 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.43984.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44682.log 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38166.log: /var/log/ceph/cluster1-client.admin.45266.log: gzip 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38166.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.29792.log 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61909.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.44682.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.45266.log.gz 2024-09-16T11:45:40.016 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32895.log 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44682.log.gz 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29792.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29792.log.gz 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.31737.log 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.32895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72602.log 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.32895.log.gz 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.31737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53817.log 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.31737.log.gz 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.72602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45647.log 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53817.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72602.log.gz 2024-09-16T11:45:40.017 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45360.log 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.45647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32086.log 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.53817.log.gz 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49594.log 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32086.log: gzip 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.45360.log.gz 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr: -5 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45647.log.gz 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.55461.log 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.49594.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32086.log.gz 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32179.log 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.49594.log.gz 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.55461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65500.log 2024-09-16T11:45:40.018 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69703.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55461.log.gz 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32179.log.gz 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36029.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.69703.log: --verbose -- /var/log/ceph/cluster2-client.admin.47613.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.36029.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69703.log.gz 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65500.log.gz 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33731.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36029.log.gz 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44212.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60761.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33731.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.47613.log.gz 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62653.log 2024-09-16T11:45:40.019 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44212.log: /var/log/ceph/cluster2-client.mirror.60761.log: gzip -5 --verbose 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.33731.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2-client.mirror.62612.log 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44212.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.66415.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60761.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40988.log 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40199.log 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61951.log 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62612.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35534.log 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.40988.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56972.log 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40199.log.gz 2024-09-16T11:45:40.020 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.35534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40018.log 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35534.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62653.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61951.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.56972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38774.log 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39951.log 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40018.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.56972.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39758.log 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.40018.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65566.log 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.39951.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38774.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.39758.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39951.log.gz 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68131.log 2024-09-16T11:45:40.021 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53996.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65566.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.39758.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.68131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72573.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68131.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65566.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35634.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71999.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.53996.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35634.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.72573.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35634.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68552.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54693.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71999.log.gz 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67955.log 2024-09-16T11:45:40.022 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36061.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54693.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68552.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61480.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.54693.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67955.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54285.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.0.36061.log: 18.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36061.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60073.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52585.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54285.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58348.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61480.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61584.log 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52585.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60073.log.gz 2024-09-16T11:45:40.023 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.58348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34974.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.52585.log.gz 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71655.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61584.log: /var/log/ceph/cluster2-client.mirror.34974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63364.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.58348.log.gz 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61584.log.gz 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68886.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67990.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66175.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67990.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68886.log.gzgzip 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.47956.log 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67990.log.gz 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34974.log.gz 2024-09-16T11:45:40.024 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73810.log 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71655.log.gz 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46806.log 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63364.log.gz 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47956.log: /var/log/ceph/cluster2-client.admin.73810.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73810.log.gz 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39880.log 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.47956.log.gz 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60689.log 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.46806.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66175.log.gz 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.39880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34420.log 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46806.log.gz 2024-09-16T11:45:40.025 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68397.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39880.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46966.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34420.log: /var/log/ceph/cluster1-client.admin.68397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47050.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34420.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46966.log: gzip -5 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.68397.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.45526.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47050.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46966.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54223.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.47050.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69232.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62510.log 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.45526.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69232.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.54223.log.gz 2024-09-16T11:45:40.026 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54419.log 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62510.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69232.log.gz 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73516.log 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52242.log 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54419.log.gz 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73516.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31381.log 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62510.log.gz 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73516.log.gz 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38706.log 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52242.log: /var/log/ceph/cluster1-client.mirror.31381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48247.log 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr: 63.7% -- replaced with /var/log/ceph/cluster1-client.mirror.31381.log.gz 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.52242.log.gz 2024-09-16T11:45:40.027 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38706.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55692.log 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68151.log 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.48247.log.gz 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.55692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71384.log 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.68151.log: --verbose -- /var/log/ceph/cluster1-client.admin.34846.log 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.55692.log.gz 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.71384.log: gzip -5 --verbose 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71384.log.gz 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.mirror.38047.log 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41573.log 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30842.log 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.34846.log.gz 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60689.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38047.log.gz 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.028 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41573.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68151.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38986.log 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41573.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52501.log 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30842.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47900.log 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38986.log: /var/log/ceph/cluster2-client.admin.52501.log: 87.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.38986.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49562.log 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.52501.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.47900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62829.log 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49428.log 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.49562.log: /var/log/ceph/cluster1-client.mirror.62829.log: gzip -5 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49562.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.51696.log 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62829.log.gz 2024-09-16T11:45:40.029 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47900.log.gz 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66676.log 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8%/var/log/ceph/cluster1-client.admin.51696.log: -- replaced with /var/log/ceph/cluster2-client.admin.49428.log.gz 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62985.log 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.51696.log.gz/var/log/ceph/cluster1-client.mirror.66676.log: 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62489.log 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66676.log.gz 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62985.log: gzip -5 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62985.log.gz --verbose 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2-client.mirror.61339.log 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51183.log 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37533.log 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62489.log.gz 2024-09-16T11:45:40.030 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38706.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.51183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38295.log 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.37533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67440.log 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.38295.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37533.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51183.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31518.log 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67440.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.38295.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61339.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71675.log 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.31518.log: -- /var/log/ceph/cluster1-client.mirror.36937.log 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.31518.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.71675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67858.log 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.32553.log.gz 2024-09-16T11:45:40.031 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.36937.log: 92.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.67440.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31765.log 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40772.log 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67858.log: gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.31765.log: /var/log/ceph/cluster2-client.mirror.48109.log 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67858.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.31765.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60524.log 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.48109.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40772.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35459.log 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71675.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47110.log 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36937.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60524.log: /var/log/ceph/cluster1-client.mirror.35459.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48109.log.gz 2024-09-16T11:45:40.032 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73046.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35459.log.gz 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60524.log.gz 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60332.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.73046.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.47110.log.gz 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35609.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73046.log.gz 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39787.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60332.log.gz 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61359.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35609.log.gz 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54786.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68735.log 2024-09-16T11:45:40.033 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.39787.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50802.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.54786.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.68735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39410.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61359.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50802.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.68735.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73642.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67265.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.39410.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.50802.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30296.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73642.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39410.log.gz 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.67265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55660.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65676.log 2024-09-16T11:45:40.034 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.67265.log.gz 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30296.log.gz 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.55660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45964.log 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55660.log.gz 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68922.log 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46435.log 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.68922.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65676.log.gz 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4%gzip -- replaced with /var/log/ceph/cluster1-client.admin.45964.log.gz -5 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.59857.log 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43552.log 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72073.log 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.46435.log.gz 2024-09-16T11:45:40.035 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.43552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35289.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59857.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43552.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59697.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.35289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44553.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.72073.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35289.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63410.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65720.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.44553.log: /var/log/ceph/cluster1-client.mirror.63410.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59697.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71053.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44553.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63410.log.gz 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66356.log 2024-09-16T11:45:40.036 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65720.log: /var/log/ceph/cluster2-client.mirror.71053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42014.log 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66356.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71053.log.gz 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63386.log 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65720.log.gz 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59627.log 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66356.log.gz 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41019.log 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42014.log.gz 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49995.log 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59627.log: /var/log/ceph/cluster1-client.admin.41019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51816.log 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63386.log.gz 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.49995.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41019.log.gz 2024-09-16T11:45:40.037 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69157.log 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59627.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49995.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36070.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32864.log 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51816.log: /var/log/ceph/cluster1-client.mirror.69157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38447.log 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51816.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69157.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.32864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45798.log 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.38447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69867.log 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.32864.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45798.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.38447.log.gz 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50865.log 2024-09-16T11:45:40.038 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31552.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69867.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.45798.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63770.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69867.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31552.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.50865.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62548.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr: 53.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31552.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63119.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr: 62.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63770.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55992.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63119.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62548.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33950.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.55992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29916.log 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63119.log.gz 2024-09-16T11:45:40.039 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55308.log 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55992.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29916.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29916.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63672.log 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30506.log 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.33950.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55308.log: /var/log/ceph/cluster2-client.mirror.63672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44588.log 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68922.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.55308.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53435.log 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30506.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.44588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55964.log 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44588.log.gz 2024-09-16T11:45:40.040 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61310.log 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.53435.log: /var/log/ceph/cluster2-client.admin.55964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67535.log 92.4% 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63672.log.gz 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73432.log 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53435.log.gz 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.55964.log.gz 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61411.log 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62762.log 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73432.log.gz 2024-09-16T11:45:40.041 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61411.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67535.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70105.log 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61310.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67296.log 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62762.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43465.log 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70105.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.67296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38944.log 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43465.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.67296.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66949.log 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61411.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57343.log 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.43465.log.gz 2024-09-16T11:45:40.042 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66949.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38944.log.gz 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56664.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66949.log.gz 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56764.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.56664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66455.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.57343.log.gz 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56664.log.gz 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.56764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32018.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37800.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.56764.log.gz 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32018.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66455.log.gz 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35559.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-16T11:45:40.043 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.35559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50155.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.37800.log.gz 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35559.log.gz 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61663.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47872.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61663.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.50155.log.gz 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49002.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61663.log.gz 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34708.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47872.log: /var/log/ceph/cluster2-client.admin.49002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29914.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.47872.log.gz 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62633.log 2024-09-16T11:45:40.044 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.49002.log.gz 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.34708.log.gz 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37860.log 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29914.log.gz 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.68927.log 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62633.log: /var/log/ceph/cluster2-client.admin.37860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30884.log 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62633.log.gz 2024-09-16T11:45:40.045 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.37860.log.gz 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.68927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52529.log 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.68927.log.gz 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62198.log 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30884.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30884.log.gz 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46026.log 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.52529.log: /var/log/ceph/cluster2-client.mirror.62198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37919.log 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33090.log 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52529.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62198.log.gz 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.37919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58733.log 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.46026.log.gz 2024-09-16T11:45:40.046 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53784.log 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.33090.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37919.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49896.log 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58733.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33090.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47522.log 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58733.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47553.log 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.53784.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32018.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47522.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.49896.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43434.log 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.47522.log.gz 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72723.log 2024-09-16T11:45:40.047 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47553.log: /var/log/ceph/cluster2-client.admin.43434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49089.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.47553.log.gz 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49124.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72723.log.gz 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.43434.log.gz 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.49089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72258.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49089.log.gz 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.49124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63599.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49124.log.gz 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67356.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37569.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.67356.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63599.log.gz 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.72258.log.gz -5 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.52091.log 2024-09-16T11:45:40.048 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37569.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.67356.log.gz 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47582.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.52091.log: 87.9%gzip -- replaced with /var/log/ceph/cluster1-client.admin.37569.log.gz -5 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.72460.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68088.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72460.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.47582.log.gz 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52091.log.gz 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57521.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.72460.log.gz 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66821.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.68088.log: /var/log/ceph/cluster2-client.admin.57521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35335.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61972.log 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.57521.log.gz 2024-09-16T11:45:40.049 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66821.log.gz 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46994.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.35335.log.gz 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62219.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.46994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56332.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62219.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61972.log.gz 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.68088.log.gz -5 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.65522.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71274.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46994.log.gz 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.56332.log: /var/log/ceph/cluster2-client.mirror.65522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41167.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62219.log.gz 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.71274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62469.log 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56332.log.gz 2024-09-16T11:45:40.050 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71274.log.gz 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41167.log: 89.3%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.65522.log.gz -5 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.51665.log 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62962.log 88.2% 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.41167.log.gz 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42684.log 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62469.log.gz 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46141.log 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.51665.log.gz 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62962.log.gz 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.42684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66798.log 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42684.log.gz 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.46141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69669.log 2024-09-16T11:45:40.051 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66798.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.46141.log.gz 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48619.log 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66798.log.gz 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49831.log 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69669.log.gz 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.48619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51970.log 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48619.log.gz 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62039.log 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71697.log 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.49831.log.gz 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62039.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.51970.log.gz 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47274.log 2024-09-16T11:45:40.052 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62039.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71918.log 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.47274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38233.log 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.71918.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47274.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71697.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31313.log 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.71918.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.38233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40230.log 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.31313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39610.log 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.38233.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr: 69.6% -- replaced with /var/log/ceph/cluster1-client.admin.31313.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35426.log 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39610.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40230.log.gz 2024-09-16T11:45:40.053 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.68913.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40424.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.39610.log.gz 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.68913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52441.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.68913.log.gz 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.40424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45058.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40424.log.gz 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44466.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60963.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45058.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35426.log.gz 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.52441.log.gz 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51533.log 2024-09-16T11:45:40.054 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.45058.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60963.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.44466.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35130.log 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60963.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66415.log 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.35130.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.51533.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44991.log 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71357.log 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.35130.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.44991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43833.log 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44991.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.71357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67506.log 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71357.log.gz 2024-09-16T11:45:40.055 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58939.log 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.43833.log: /var/log/ceph/cluster2-client.mirror.67506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66107.log 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56484.log 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43833.log.gz 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58939.log.gz 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47462.log 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.56484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.66443.log 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.56484.log.gz 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.47462.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66107.log.gz 2024-09-16T11:45:40.056 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31603.log 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr: 90.9%/var/log/ceph/cluster1-client.mirror.1.66443.log: -- replaced with /var/log/ceph/cluster2-client.mirror.67506.log.gz 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65632.log 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67796.log 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31603.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47462.log.gz 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr: 64.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31603.log.gz 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35998.log 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.67796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51468.log 2024-09-16T11:45:40.057 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.35998.log: 87.3%gzip -5 89.2% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.65632.log.gz /var/log/ceph/cluster1-client.mirror.70022.log 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67796.log.gz 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.51468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63552.log 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35998.log.gz 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70022.log: 87.4%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.51468.log.gz 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.45295.log 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70022.log.gz 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36851.log 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63552.log: /var/log/ceph/cluster2-client.admin.45295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59514.log 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63552.log.gz 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.45295.log.gz 2024-09-16T11:45:40.058 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.36851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41485.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38867.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59514.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.36851.log.gz 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.41485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62261.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38867.log: gzip -5 --verbose 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59514.log.gz 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2-client.admin.55193.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38867.log.gz 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41485.log.gz 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65896.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.55193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62155.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.55193.log.gz 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67655.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62110.log 2024-09-16T11:45:40.059 INFO:teuthology.orchestra.run.smithi204.stderr: 89.4%/var/log/ceph/cluster2-client.mirror.67655.log: -- replaced with /var/log/ceph/cluster2-client.mirror.65896.log.gz 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34026.log 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67655.log.gz 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55433.log 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62110.log: /var/log/ceph/cluster1-client.admin.34026.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62110.log.gz 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.34026.log.gz 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47712.log 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.55433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57125.log 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.47712.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.55433.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72664.log 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47712.log.gz 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.57125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42573.log 2024-09-16T11:45:40.060 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.72664.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57125.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63464.log 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72664.log.gz 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.42573.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.42573.log.gz 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53658.log 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63064.log 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63464.log.gz 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.53658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52844.log 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67156.log 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53658.log.gz 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63064.log: /var/log/ceph/cluster2-client.mirror.52844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47374.log 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63064.log.gz 2024-09-16T11:45:40.061 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52844.log.gz/var/log/ceph/cluster2-client.mirror.67156.log: 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38330.log 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62155.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62261.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.66443.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67156.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.47374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40652.log 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.38330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.31723.log 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.47374.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.38330.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30968.log 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40652.log.gz 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33424.log 2024-09-16T11:45:40.062 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.31723.log: 18.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.31723.log.gz 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48779.log 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30968.log.gz 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59459.log 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48779.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33424.log.gz 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73168.log 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59459.log: 87.6%gzip -- replaced with /var/log/ceph/cluster2-client.admin.48779.log.gz -5 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.71299.log 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73168.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73168.log.gz 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66990.log 2024-09-16T11:45:40.063 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70513.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71299.log: /var/log/ceph/cluster2-client.mirror.66990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37094.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71299.log.gz 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66990.log.gz 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.70513.log: /var/log/ceph/cluster2-client.mirror.1.72699.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59459.log.gz 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69363.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.70513.log.gz 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.37094.log.gz 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.1.72699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70319.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34677.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.68922.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66380.log 2024-09-16T11:45:40.064 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3%/var/log/ceph/cluster1-client.mirror.0.68922.log: -- replaced with /var/log/ceph/cluster2-client.mirror.70319.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39821.log 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.72699.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69363.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.34677.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54191.log 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66415.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39821.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66380.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66086.log 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.54191.log: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73936.log 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.39821.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54191.log.gz 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48215.log 2024-09-16T11:45:40.065 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36306.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73936.log.gz 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66086.log.gz 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.48215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59717.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.36306.log: 86.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48215.log.gz 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59196.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.59717.log: gzip 87.9% -5 -- replaced with /var/log/ceph/cluster1-client.admin.36306.log.gz 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.36552.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.mirror.59196.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35006.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41636.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36552.log.gz 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70482.log 2024-09-16T11:45:40.066 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63433.log 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59196.log.gz 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35006.log.gz 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.70482.log: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59717.log.gz 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.41636.log.gz 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36645.log 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.70482.log.gz 2024-09-16T11:45:40.067 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.mirror.63433.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66622.log 2024-09-16T11:45:40.068 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63433.log.gz 2024-09-16T11:45:40.068 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43193.log 2024-09-16T11:45:40.068 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36645.log.gz 2024-09-16T11:45:40.068 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41138.log 2024-09-16T11:45:40.068 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43193.log: gzip 87.2% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.66622.log.gz --verbose 2024-09-16T11:45:40.068 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2.audit.log 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36075.log 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.43193.log.gz 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2.audit.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41138.log.gz 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31732.log 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.1.36075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59673.log 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr: 17.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36075.log.gz 2024-09-16T11:45:40.069 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66151.log 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49189.log 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48719.log 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71833.log 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8%/var/log/ceph/cluster1-client.admin.48719.log: -- replaced with /var/log/ceph/cluster2-client.admin.49189.log.gz 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59673.log.gz 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60146.log 2024-09-16T11:45:40.070 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48719.log.gz 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44315.log 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66151.log.gz 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42788.log 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71833.log.gz 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60146.log.gz 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68770.log 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.44315.log: /var/log/ceph/cluster2-client.admin.42788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46201.log 2024-09-16T11:45:40.071 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44315.log.gz 2024-09-16T11:45:40.072 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.72669.log 2024-09-16T11:45:40.072 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.42788.log.gz 2024-09-16T11:45:40.072 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.68770.log: /var/log/ceph/cluster2-client.admin.46201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56424.log 2024-09-16T11:45:40.072 INFO:teuthology.orchestra.run.smithi204.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31732.log.gz 2024-09-16T11:45:40.072 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.1.72669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46057.log 2024-09-16T11:45:40.072 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% 92.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-16T11:45:40.073 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.72669.log.gz 2024-09-16T11:45:40.073 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.46201.log.gz 2024-09-16T11:45:40.073 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43953.log 2024-09-16T11:45:40.073 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56424.log: /var/log/ceph/cluster2-client.admin.46057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42967.log 2024-09-16T11:45:40.073 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.56424.log.gz 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.46057.log.gz 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61754.log 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43953.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68770.log.gz 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40859.log 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.43953.log.gz 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35161.log 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42967.log.gz 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.40859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50924.log 2024-09-16T11:45:40.074 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40859.log.gz 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.35161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52909.log 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.50924.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.35161.log.gz 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29545.log 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58826.log 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50924.log.gz 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29545.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.admin.29545.log.gz -- 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr: /var/log/ceph/cluster2-client.mirror.0.72664.log 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.52909.log.gz 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31279.log 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.0.72664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70734.log 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58826.log.gz 2024-09-16T11:45:40.075 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.31279.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.31279.log.gz 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33246.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72622.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70734.log.gz 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45707.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.72622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45235.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3%/var/log/ceph/cluster1-client.admin.45707.log: -- replaced with /var/log/ceph/cluster1-client.admin.33246.log.gz 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.72664.log.gz 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65951.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.45707.log.gz 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.admin.45235.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60600.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.65951.log: gzip -5 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45235.log.gz 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.31535.log 2024-09-16T11:45:40.076 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65951.log.gz 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61866.log 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72622.log.gz 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.31535.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60600.log.gz 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 69.5% -- replaced with /var/log/ceph/cluster2-client.admin.31535.log.gz 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2-client.mirror.32525.log 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61866.log: gzip -5 --verbose -- 84.4% /var/log/ceph/cluster2-client.admin.54316.log 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61866.log.gz 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40318.log 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54316.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61754.log.gz 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36757.log 2024-09-16T11:45:40.077 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50030.log 2024-09-16T11:45:40.078 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.40318.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.54316.log.gz 2024-09-16T11:45:40.078 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63923.log 2024-09-16T11:45:40.078 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40318.log.gz 2024-09-16T11:45:40.078 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36757.log.gz 2024-09-16T11:45:40.078 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44892.log 2024-09-16T11:45:40.079 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.50030.log: /var/log/ceph/cluster2-client.mirror.63923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57975.log 2024-09-16T11:45:40.079 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50030.log.gz 2024-09-16T11:45:40.079 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67460.log 2024-09-16T11:45:40.079 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.57975.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.44892.log.gz 2024-09-16T11:45:40.079 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66062.log 2024-09-16T11:45:40.080 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57975.log.gz 2024-09-16T11:45:40.080 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.mirror.67460.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69894.log 2024-09-16T11:45:40.080 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44083.log 2024-09-16T11:45:40.080 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69894.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66062.log.gz 2024-09-16T11:45:40.080 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30590.log 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69894.log.gz 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.44083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69976.log 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44083.log.gz 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67460.log.gz 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30590.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30590.log.gz 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61528.log 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35655.log 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69976.log: /var/log/ceph/cluster2-client.mirror.61528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30338.log 86.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69976.log.gz 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34057.log 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35655.log.gz 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30338.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59538.log 2024-09-16T11:45:40.081 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.30338.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58753.log 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37600.log 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59538.log: 90.3%/var/log/ceph/cluster2-client.mirror.58753.log: -- replaced with /var/log/ceph/cluster1-client.admin.34057.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34451.log 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59538.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37600.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.37600.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50835.log 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34451.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58753.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38266.log 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63923.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50835.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.34451.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.50835.log.gz 2024-09-16T11:45:40.082 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61706.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.38266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73894.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61706.log: gzip -5 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.38266.log.gz 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.51030.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73894.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53407.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.73894.log.gz 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.51030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48882.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61706.log.gz 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69328.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51030.log.gz 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.53407.log.gz 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48882.log: -- /var/log/ceph/cluster1-client.admin.56393.log 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.69328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33362.log 87.7% 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.48882.log.gz 2024-09-16T11:45:40.083 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56393.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.69328.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61528.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.admin.50336.log 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.56393.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45897.log 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50336.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33362.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58442.log 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.45897.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.50336.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70689.log 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45897.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.58442.log: -- /var/log/ceph/cluster2-client.mirror.63634.log 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70689.log: gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70689.log.gz 2024-09-16T11:45:40.084 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.mirror.72236.log 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58442.log.gz 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44620.log 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63634.log.gz 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42105.log 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55532.log 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44620.log: /var/log/ceph/cluster2-client.admin.42105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68790.log 2024-09-16T11:45:40.085 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.44620.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.55532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53088.log 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.42105.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42936.log 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55532.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72236.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68790.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.68922.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67821.log 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55339.log 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.53088.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67821.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42936.log.gz 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68483.log 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58010.log 2024-09-16T11:45:40.086 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55339.log: /var/log/ceph/cluster1-client.mirror.68483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72175.log 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68790.log.gz 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55339.log.gz 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68309.log 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58010.log: /var/log/ceph/cluster1-client.admin.72175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37356.log 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68309.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68209.log 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58010.log.gz 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.72175.log.gz 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37356.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.68309.log.gz 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55933.log 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68483.log.gz 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.37356.log.gzgzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.68209.log: /var/log/ceph/cluster2-client.mirror.57664.log 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.087 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.55933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52063.log 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.68209.log.gz 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.57664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57458.log 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.55933.log.gz 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52063.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57664.log.gz 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37718.log 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.52063.log.gz 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.57458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63031.log 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44183.log 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.37718.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.57458.log.gz 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58379.log 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63031.log.gz 2024-09-16T11:45:40.088 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30926.log 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.58379.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34739.log 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44183.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59261.log 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.58379.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37718.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30926.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60189.log 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53376.log 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.34739.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69751.log 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59261.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60189.log.gz 2024-09-16T11:45:40.089 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63519.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.69751.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.53376.log.gz 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43587.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67821.log.gz 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.69751.log.gz 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40577.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.43587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71146.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63519.log.gz 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43587.log.gz 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.40577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54344.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.71146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42508.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40577.log.gz 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.54344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68626.log 2024-09-16T11:45:40.090 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43493.log 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.42508.log: /var/log/ceph/cluster1-client.mirror.68626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70288.log 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54344.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.42508.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68626.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.43493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36789.log 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71146.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70288.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43493.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.admin.51243.log 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.36789.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70288.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45459.log 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.36789.log.gz 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62705.log 2024-09-16T11:45:40.091 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.45459.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.51243.log.gz 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46294.log 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45459.log.gz 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36340.log 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62705.log: /var/log/ceph/cluster2-client.mirror.46294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38385.log 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62705.log.gz 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.36340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59161.log 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46294.log.gz 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32472.log 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.36340.log.gz 2024-09-16T11:45:40.092 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.59161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68241.log 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50396.log 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.59161.log.gz 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32472.log.gz 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45619.log 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68241.log.gz 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38385.log.gz 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42448.log 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45619.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.50396.log.gzgzip 2024-09-16T11:45:40.093 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.41457.log 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46400.log 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42448.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.45619.log.gz 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62134.log 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42448.log.gz 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46172.log 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.46400.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.41457.log.gz 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73600.log 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46400.log.gz 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62134.log.gz 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.46172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41364.log 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73600.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.46172.log.gz 2024-09-16T11:45:40.094 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73600.log.gz 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46735.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48180.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41364.log: /var/log/ceph/cluster1-client.mirror.46735.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60450.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.48180.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.41364.log.gz 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62240.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46735.log.gz 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48180.log.gz 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70213.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62240.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60450.log.gz 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50896.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44920.log 2024-09-16T11:45:40.095 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62240.log.gz 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32144.log 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.44920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36882.log 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.50896.log.gz 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.mirror.32144.log: -5 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44920.log.gz 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.65698.log 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.36882.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32144.log.gz 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70213.log.gz 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.31760.log 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36882.log.gz 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56943.log 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.31760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49715.log 2024-09-16T11:45:40.096 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56943.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65698.log.gz 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42204.log 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.49715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36371.log 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.56943.log.gz 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.42204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.66406.log 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42204.log.gz 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.36371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42755.log 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49715.log.gz 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46935.log 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.66406.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36371.log.gz 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.66406.log.gz 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.42755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54156.log 2024-09-16T11:45:40.097 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46935.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.42755.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50367.log 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.31760.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.46935.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45738.log 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.54156.log: /var/log/ceph/cluster1-client.admin.50367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69069.log 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54156.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45738.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.50367.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67610.log 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42996.log 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69069.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.45738.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.67610.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69069.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46873.log 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr: 86.3% -- replaced with /var/log/ceph/cluster1-client.mirror.67610.log.gz 2024-09-16T11:45:40.098 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43255.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.42996.log: /var/log/ceph/cluster1-client.admin.46873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39291.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.42996.log.gz 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43255.log: 88.4%gzip -- replaced with /var/log/ceph/cluster1-client.admin.46873.log.gz 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72997.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.39291.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.43255.log.gz 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37630.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.39291.log.gz 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49687.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.72997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72997.log.gz 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51155.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52876.log 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.37630.log.gz 2024-09-16T11:45:40.099 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.49687.log.gz 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69603.log 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65852.log 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52876.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.51155.log.gz 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50248.log 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.admin.52876.log.gz 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69603.log.gz 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41047.log 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44494.log 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.50248.log.gz 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.41047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51785.log 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65852.log.gz 2024-09-16T11:45:40.100 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34877.log 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.44494.log: /var/log/ceph/cluster2-client.admin.51785.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41047.log.gz 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.32525.log 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44494.log.gz 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72831.log 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34877.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.51785.log.gz 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70652.log 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.0.32525.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34877.log.gz 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37182.log 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72831.log.gz 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61437.log 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70652.log.gz 2024-09-16T11:45:40.101 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.37182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31432.log 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31569.log 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61437.log.gz 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37182.log.gz 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.31432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69568.log 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31432.log.gz 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58169.log 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31569.log.gz 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73852.log 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69568.log.gz 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57731.log 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58169.log.gz 2024-09-16T11:45:40.102 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50455.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73852.log.gz 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37698.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57731.log: /var/log/ceph/cluster1-client.mirror.50455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.57731.log.gz 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.37698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70711.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50455.log.gz 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51124.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70711.log: gzip -5 --verbose -- 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70711.log.gz 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr: /var/log/ceph/cluster1-client.mirror.72922.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51124.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37698.log.gz 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70784.log 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.51124.log.gz 2024-09-16T11:45:40.103 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72922.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39852.log 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72922.log.gz 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58072.log 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39852.log: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30422.log 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70784.log.gz 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40800.log 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.39852.log.gz 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58072.log.gz 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.45826.log/var/log/ceph/cluster1-client.admin.30422.log: 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30422.log.gz 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.40800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34954.log 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40800.log.gz 2024-09-16T11:45:40.104 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.45826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30800.log 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.34954.log: --verbose -- /var/log/ceph/cluster1-client.admin.33793.log 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45826.log.gz 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.30800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30800.log.gz 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34954.log.gz 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51274.log 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59995.log 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.51274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63847.log 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.33793.log.gz 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59995.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.51274.log.gz 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72404.log 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59995.log.gz 2024-09-16T11:45:40.105 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63847.log: gzip -5 --verbose -- 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63847.log.gz 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr: /var/log/ceph/cluster1-client.admin.43374.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65588.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72404.log.gz 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47175.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.43374.log.gz 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60362.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5/var/log/ceph/cluster2-client.admin.47175.log: --verbose -- /var/log/ceph/cluster2-client.mirror.32553.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60362.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65588.log.gz 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 87.6% --verbose -- -- replaced with /var/log/ceph/cluster2-client.admin.47175.log.gz 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr: /var/log/ceph/cluster2-client.admin.46468.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60362.log.gz 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69130.log 2024-09-16T11:45:40.106 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32553.log: /var/log/ceph/cluster2-client.admin.46468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46235.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38739.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.46468.log.gz 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69130.log.gz 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46235.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.46235.log.gz 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.admin.43405.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49030.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38739.log.gz 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33215.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.49030.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43405.log.gz 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31586.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49030.log.gz 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.39382.log 2024-09-16T11:45:40.107 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31586.log: /var/log/ceph/cluster1-client.admin.33215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73684.log 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31586.log.gz 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.33215.log.gz 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29225.log 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73684.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.admin.41198.log 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.73684.log.gz 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.39382.log.gz 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.admin.29225.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.55754.log 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29225.log.gz 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58868.log 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55754.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.41198.log.gz 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72788.log 2024-09-16T11:45:40.108 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64487.log 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58868.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.55754.log.gz 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.72788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60029.log 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72788.log.gz 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.64487.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58868.log.gz 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60668.log 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64487.log.gz 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56512.log 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.60668.log: -- replaced with /var/log/ceph/cluster1-client.mirror.60029.log.gz 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65455.log 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.56512.log: -- /var/log/ceph/cluster2-client.mirror.36459.log 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60668.log.gz 2024-09-16T11:45:40.109 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65455.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56512.log.gz 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36103.log 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6%/var/log/ceph/cluster2-client.mirror.36459.log: -- replaced with /var/log/ceph/cluster2-client.mirror.65455.log.gz 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66443.log 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48971.log 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36459.log.gz 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.1.36103.log: /var/log/ceph/cluster2-client.mirror.66443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40259.log 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48971.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.48971.log.gz 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45388.log 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40259.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.40259.log.gz 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31475.log 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.45388.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45388.log.gz 2024-09-16T11:45:40.110 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67486.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.31475.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31475.log.gz 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33303.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr: 93.3%/var/log/ceph/cluster2-client.mirror.67486.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66443.log.gz 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52270.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67486.log.gz 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.33303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48460.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.52270.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52270.log.gz 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41297.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.33303.log.gz 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62918.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.48460.log.gz/var/log/ceph/cluster1-client.mirror.41297.log: 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41297.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41866.log 2024-09-16T11:45:40.111 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29126.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41866.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62918.log.gz 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59792.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.41866.log.gz 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29126.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29126.log.gz 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61684.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62362.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61684.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59792.log.gz 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49337.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62362.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62362.log.gz 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72972.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.49337.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.49337.log.gz 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31620.log 2024-09-16T11:45:40.112 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.72972.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72972.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53599.log 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31620.log: 56.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31620.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51903.log 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53599.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.53599.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64335.log 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.51903.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51903.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63830.log 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.64335.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64335.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44435.log 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63830.log: 63.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63830.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61684.log.gz 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52470.log 2024-09-16T11:45:40.113 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66751.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44435.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68527.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66751.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.52470.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49865.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66751.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72429.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49865.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68527.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.49865.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45091.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49627.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45091.log: /var/log/ceph/cluster1-client.mirror.72429.log: 87.8% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.45091.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72429.log.gz 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73390.log 2024-09-16T11:45:40.114 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.49627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38507.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49627.log.gz/var/log/ceph/cluster2-client.admin.73390.log: 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73390.log.gz 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46497.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.38507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47403.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.38507.log.gz 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.46497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67066.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47403.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.46497.log.gz 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45588.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.47403.log.gz 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47203.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45588.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.45588.log.gz 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33393.log 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.47203.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47203.log.gz 2024-09-16T11:45:40.115 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73144.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33393.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.33393.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37029.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.73144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73144.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38602.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37029.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.37029.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48654.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.38602.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.38602.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70362.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.48654.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48654.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66968.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70362.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67066.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65742.log 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66968.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66968.log.gz 2024-09-16T11:45:40.116 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59653.log 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65742.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70362.log.gz 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48016.log 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65742.log.gz 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54448.log 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48016.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59653.log.gz 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.48016.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55099.log 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53057.log 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54448.log: /var/log/ceph/cluster1-client.mirror.55099.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55099.log.gz 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54448.log.gz 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62336.log 2024-09-16T11:45:40.117 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51334.log 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.53057.log.gz/var/log/ceph/cluster1-client.mirror.62336.log: 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43027.log 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51334.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62336.log.gz 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43774.log 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.51334.log.gz 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51303.log 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.43774.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.43027.log.gz 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31398.log 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.43774.log.gz 2024-09-16T11:45:40.118 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54388.log 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51303.log: /var/log/ceph/cluster1-client.mirror.31398.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.31398.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.51303.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50771.log 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35071.log 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.54388.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47641.log 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.35071.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.50771.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65764.log 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.35071.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.47641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70233.log 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65764.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47641.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32553.log.gz 2024-09-16T11:45:40.119 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70049.log 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65764.log.gz 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63090.log 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46676.log 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70049.log: /var/log/ceph/cluster1-client.mirror.63090.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70049.log.gz 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63090.log.gz 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43680.log 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70233.log.gz 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.46906.log 2024-09-16T11:45:40.120 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.46676.log: /var/log/ceph/cluster2-client.mirror.43680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58685.log 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.46676.log.gz 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43680.log.gz 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.46906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31010.log 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.58685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72208.log 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.46906.log.gz 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.31010.log: 0.0% 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58685.log.gz 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.31010.log.gz 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52679.log 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48340.log 2024-09-16T11:45:40.121 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52679.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.72208.log.gz 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27418.log 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48340.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.52679.log.gz 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50527.log 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55370.log 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.48340.log.gz 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.tmp-client.admin.27418.log: /var/log/ceph/cluster1-client.mirror.50527.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27418.log.gz 2024-09-16T11:45:40.122 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.32525.log.gz -5 --verbose -- /var/log/ceph/cluster1-client.admin.49218.log 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50527.log.gz 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.55370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56571.log 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.55370.log.gz 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr: --/var/log/ceph/cluster1-client.admin.49218.log: /var/log/ceph/cluster1-client.admin.33562.log 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32493.log 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.49218.log.gz/var/log/ceph/cluster2-client.mirror.56571.log: 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36103.log.gz 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56571.log.gz 2024-09-16T11:45:40.123 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.33562.log: --verbose -- /var/log/ceph/cluster1-client.admin.53347.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56244.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53347.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.33562.log.gz 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69999.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.53347.log.gz 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64609.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69999.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.56244.log.gz 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69999.log.gz 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.53717.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.64609.log: gzip -5 --verbose 86.5% -- /var/log/ceph/cluster1-client.admin.59359.log -- replaced with /var/log/ceph/cluster1-client.mirror.64609.log.gz 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.53717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33700.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53717.log.gz 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.59359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65654.log 2024-09-16T11:45:40.124 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.admin.33700.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39185.log 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32401.log 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33700.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.admin.59359.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.39185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33283.log 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65654.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.32401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62449.log 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.33283.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.32401.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30148.log 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62449.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39185.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67088.log 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62449.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30148.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30148.log.gz 2024-09-16T11:45:40.125 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31330.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33283.log.gz 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40894.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.31330.log: gzip -5 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31330.log.gz 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.33148.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.40894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40894.log.gz 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.33148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46556.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33148.log.gz 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53282.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-mon.a.log: /var/log/ceph/cluster2-client.mirror.46556.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32493.log.gz 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33888.log 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62307.log/var/log/ceph/cluster1-client.mirror.53282.log: 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46556.log.gz 2024-09-16T11:45:40.126 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4%/var/log/ceph/cluster1-client.admin.33888.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53282.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32242.log 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58640.log 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.33888.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62307.log: /var/log/ceph/cluster1-client.admin.32242.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62307.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.32242.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62568.log 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60288.log 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.58640.log: /var/log/ceph/cluster2-client.mirror.62568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60858.log 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60288.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60288.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71018.log 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60858.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62568.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60858.log.gz 2024-09-16T11:45:40.127 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68359.log 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67088.log.gz 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44832.log 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57828.log 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71018.log.gz 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68359.log: /var/log/ceph/cluster1-client.admin.44832.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68359.log.gz 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.44832.log.gz 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38832.log 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.57828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43126.log 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.38832.log: -- replaced with /var/log/ceph/cluster1-client.admin.57828.log.gz 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35236.log 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38832.log.gz 2024-09-16T11:45:40.128 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.43126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56731.log 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43126.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36400.log 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35236.log: /var/log/ceph/cluster2-client.admin.56731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42719.log 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.36400.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35236.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.56731.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63741.log 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.36400.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.mirror.42719.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58285.log 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.63741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61106.log 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42719.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.63741.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58640.log.gz 2024-09-16T11:45:40.129 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.66420.log 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58285.log.gz 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55725.log 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61106.log.gz 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.66420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39057.log 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr: 18.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.66420.log.gz 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5/var/log/ceph/cluster2-client.admin.55725.log: --verbose -- /var/log/ceph/cluster1-client.mirror.65984.log 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.39057.log: gzip -5 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.55725.log.gz 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.64536.log 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.39057.log.gz 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.65984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70175.log 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.64536.log: -- replaced with /var/log/ceph/cluster1-client.mirror.65984.log.gz 2024-09-16T11:45:40.130 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46365.log 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64536.log.gz 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59747.log 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70175.log.gz 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.46365.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69259.log 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59747.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46365.log.gz 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64511.log 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53479.log 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59747.log.gz 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.64511.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69259.log.gz 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64511.log.gz 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71887.log 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44118.log 2024-09-16T11:45:40.131 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.53479.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.71887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72375.log 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.mirror.44118.log: -5 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.71887.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.36110.log 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72375.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44118.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68332.log 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.72375.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62068.log 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31364.log 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68332.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0%/var/log/ceph/cluster1-client.mirror.62068.log: -- replaced with /var/log/ceph/cluster2-client.mirror.36110.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32965.log 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62068.log.gz 2024-09-16T11:45:40.132 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.31364.log: gzip -5 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31364.log.gz 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.44375.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46647.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53877.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.44375.log.gz 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.46647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46085.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%/var/log/ceph/cluster1-client.admin.53877.log: -- replaced with /var/log/ceph/cluster1-client.admin.46647.log.gz 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32965.log.gz -5 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.39669.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.53877.log.gz 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41954.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.46085.log: /var/log/ceph/cluster1-client.mirror.39669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55224.log 2024-09-16T11:45:40.133 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.41954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67588.log 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46085.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39669.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.55224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63306.log 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41954.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.67588.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.55224.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59974.log 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67588.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43286.log 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63306.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69206.log 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40926.log 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59974.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69206.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.43286.log.gz 2024-09-16T11:45:40.134 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61173.log 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69206.log.gz 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35966.log 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40926.log: /var/log/ceph/cluster1-client.admin.61173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65610.log 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.40926.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45767.log 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.61173.log.gz 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35966.log: /var/log/ceph/cluster2-client.mirror.65610.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35966.log.gz 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65610.log.gz 2024-09-16T11:45:40.135 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47779.log 2024-09-16T11:45:40.136 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-16T11:45:40.136 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.45767.log: /var/log/ceph/cluster1-client.admin.47779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31760.log 2024-09-16T11:45:40.136 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.admin.45767.log.gz 2024-09-16T11:45:40.136 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-osd.1.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.47779.log.gz 2024-09-16T11:45:40.136 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70937.log 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.31760.log: -- /var/log/ceph/cluster1-client.mirror.70545.log 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr: 82.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31760.log.gz 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34214.log 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68428.log 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70937.log.gz 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70545.log: /var/log/ceph/cluster1-client.admin.34214.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70545.log.gz 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36103.log 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.34214.log.gz 2024-09-16T11:45:40.137 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.68428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43161.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.36103.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.68428.log.gz 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33762.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60625.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.43161.log: /var/log/ceph/cluster1-client.admin.33762.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43161.log.gz 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34908.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.33762.log.gz 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40489.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60625.log.gz 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34908.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69459.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40489.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.34908.log.gz 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50276.log 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40489.log.gz 2024-09-16T11:45:40.138 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65544.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69459.log.gz 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.50276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53752.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65544.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50276.log.gz 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36103.log.gz 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68286.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65544.log.gz 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.53752.log: /var/log/ceph/cluster1-client.admin.30632.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68286.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53752.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.68950.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68286.log.gz 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster1-client.admin.30632.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58891.log 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30632.log.gz 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.1.68950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54934.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66017.log 2024-09-16T11:45:40.139 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54934.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.54934.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51562.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66017.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66017.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56211.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51562.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.51562.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58513.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56211.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.56211.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56051.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.58513.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.58513.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62784.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.56051.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56051.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56852.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62784.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62784.log.gz 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63717.log 2024-09-16T11:45:40.140 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.56852.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56852.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54256.log 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.63717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63717.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61789.log 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54256.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.54256.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35931.log 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61789.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61789.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68053.log 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58891.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35931.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35931.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72286.log 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57699.log 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.68053.log: /var/log/ceph/cluster1-client.admin.72286.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.68053.log.gz 2024-09-16T11:45:40.141 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.72286.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68022.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.57699.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57699.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33919.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57859.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.68022.log: /var/log/ceph/cluster1-client.admin.33919.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.68022.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.33919.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67675.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.57859.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57859.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58317.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47434.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67675.log: /var/log/ceph/cluster1-client.admin.58317.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.58317.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67675.log.gz 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30758.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44653.log 2024-09-16T11:45:40.142 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.47434.log.gz 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.30758.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30758.log.gz 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48520.log 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61639.log 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44653.log: /var/log/ceph/cluster2-client.admin.48520.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.44653.log.gz 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.48520.log.gz 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60928.log 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61639.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61639.log.gz 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59296.log 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70340.log 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60928.log: /var/log/ceph/cluster2-client.mirror.59296.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60928.log.gz 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45997.log 2024-09-16T11:45:40.143 INFO:teuthology.orchestra.run.smithi204.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59296.log.gz 2024-09-16T11:45:40.238 INFO:teuthology.orchestra.run.smithi053.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-16T11:45:40.300 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70340.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70340.log.gz 2024-09-16T11:45:40.300 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34119.log 2024-09-16T11:45:40.300 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32051.log 2024-09-16T11:45:40.300 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.45997.log: /var/log/ceph/cluster1-client.admin.34119.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.45997.log.gz 2024-09-16T11:45:40.300 INFO:teuthology.orchestra.run.smithi204.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.68950.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35189.log 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.admin.34119.log.gz 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32051.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32051.log.gz 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69638.log 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38407.log 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35189.log: gzip -5/var/log/ceph/cluster2-client.mirror.69638.log: --verbose -- /var/log/ceph/cluster1-client.mirror.47747.log 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69638.log.gz 2024-09-16T11:45:40.301 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38407.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35189.log.gz 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71242.log 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.47747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61290.log 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71242.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38407.log.gz 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47747.log.gz 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71242.log.gz 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2-client.mirror.71799.log 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54539.log 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56606.log 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71799.log.gz 2024-09-16T11:45:40.302 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39548.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4%/var/log/ceph/cluster2-client.mirror.56606.log: -- replaced with /var/log/ceph/cluster2-client.mirror.61290.log.gz 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.54539.log.gz 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56606.log.gz 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63787.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.39548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60823.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.63787.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.39548.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63787.log.gz 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31501.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32271.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53510.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60823.log.gz 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.31501.log: 80.0%/var/log/ceph/cluster2-client.admin.32271.log: -- replaced with /var/log/ceph/cluster2-client.admin.31501.log.gz 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32985.log 2024-09-16T11:45:40.303 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53510.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.admin.32271.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71547.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43893.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.53510.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.32985.log: /var/log/ceph/cluster2-client.mirror.71547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48686.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3%/var/log/ceph/cluster1-client.admin.43893.log: -- replaced with /var/log/ceph/cluster2-client.mirror.71547.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43893.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70570.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.48686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67722.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: 88.5%/var/log/ceph/cluster1-client.mirror.70570.log: -- replaced with /var/log/ceph/cluster1-client.admin.48686.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32985.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70570.log.gz 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster2-client.mirror.35838.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38201.log 2024-09-16T11:45:40.304 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67722.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62403.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35838.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68950.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38201.log: /var/log/ceph/cluster2-client.mirror.62403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48939.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.68950.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38201.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28913.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62403.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63881.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.28913.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28913.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.48939.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64370.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49368.log 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63881.log.gz 2024-09-16T11:45:40.305 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.64370.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50687.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.49368.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64370.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52649.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49368.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36722.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.52649.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.50687.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.52649.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43745.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.36722.log: /var/log/ceph/cluster2-client.admin.57003.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36722.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33183.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57003.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.43745.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.57003.log.gz 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71126.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.33183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50594.log 2024-09-16T11:45:40.306 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.33183.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33055.log 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.50594.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.50594.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62386.log 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.33055.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33055.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71126.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68263.log 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40741.log 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62386.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.68263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55904.log 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40741.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68263.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.admin.40741.log.gz 2024-09-16T11:45:40.307 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45932.log 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39117.log 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55904.log: /var/log/ceph/cluster1-client.mirror.45932.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55904.log.gz 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45932.log.gz 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51095.log 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50427.log 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.39117.log: /var/log/ceph/cluster2-client.admin.51095.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.admin.39117.log.gz 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.51095.log.gz 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45151.log 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42045.log 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.50427.log: /var/log/ceph/cluster2-client.admin.45151.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.50427.log.gz 2024-09-16T11:45:40.308 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.45151.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67325.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35314.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42045.log: /var/log/ceph/cluster1-client.admin.67325.log: 87.8% 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.67325.log.gz 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.42045.log.gz 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68950.log.gz 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54084.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.35314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48489.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35314.log.gz 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.54084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50562.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54084.log.gz 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr: -5/var/log/ceph/cluster2-client.admin.48489.log: --verbose -- /var/log/ceph/cluster2-client.mirror.66129.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41107.log 2024-09-16T11:45:40.309 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.50562.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.48489.log.gz 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66129.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50562.log.gz 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52032.log 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60502.log 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66129.log.gz 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.41107.log.gz 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40621.log 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44243.log 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.52032.log.gz 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.40621.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60502.log.gz 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69953.log 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57283.log 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.44243.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.40621.log.gz 2024-09-16T11:45:40.310 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.69953.log: gzip 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.44243.log.gz 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.56304.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.57283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62897.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69953.log.gz 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.56304.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.57283.log.gz 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55252.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.56304.log.gz 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39581.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62897.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62897.log.gzgzip 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.58412.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.55252.log: /var/log/ceph/cluster1-client.admin.39581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59833.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.58412.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.39581.log.gz 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55252.log.gz 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73726.log 2024-09-16T11:45:40.311 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.59833.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.admin.58412.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52003.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62177.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73726.log: /var/log/ceph/cluster2-client.admin.52003.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73726.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59833.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61727.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.52003.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61930.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62177.log: /var/log/ceph/cluster2-client.mirror.61727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32766.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61930.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62177.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61727.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39727.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.32766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57552.log 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61930.log.gz 2024-09-16T11:45:40.312 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5%/var/log/ceph/cluster1-client.admin.39727.log: -- replaced with /var/log/ceph/cluster2-client.mirror.32766.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50995.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.39727.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42074.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57552.log: /var/log/ceph/cluster2-client.mirror.50995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30041.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.42074.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50995.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.57552.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.42074.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58804.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30041.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30041.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67563.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50088.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.67563.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71617.log 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67563.log.gz 2024-09-16T11:45:40.313 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.50088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51062.log 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71617.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50088.log.gz 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71617.log.gz 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37979.log 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47987.log 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4%/var/log/ceph/cluster1-client.mirror.37979.log: -- replaced with /var/log/ceph/cluster2-client.admin.51062.log.gz 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38909.log 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.47987.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.47987.log.gz 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60123.log 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.38909.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38909.log.gz 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60646.log 2024-09-16T11:45:40.314 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60123.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60123.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44012.log 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60646.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58804.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44271.log 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.44012.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44012.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42271.log 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.44271.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44271.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58785.log 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37979.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.42271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52938.log 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58785.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.42271.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58785.log.gz 2024-09-16T11:45:40.315 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60051.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57183.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9%/var/log/ceph/cluster1-client.mirror.60051.log: -- replaced with /var/log/ceph/cluster1-client.admin.52938.log.gz 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48911.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60051.log.gz 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60646.log.gz 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61044.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.57183.log: /var/log/ceph/cluster2-client.admin.48911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46841.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57183.log.gz 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61044.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.48911.log.gz 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31671.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61044.log.gz 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37123.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.46841.log: /var/log/ceph/cluster1-client.admin.31671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47309.log 2024-09-16T11:45:40.316 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31671.log.gz 87.5% 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46841.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36587.log 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.47309.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.admin.37123.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51362.log 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47309.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69782.log 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36587.log: /var/log/ceph/cluster1-client.mirror.51362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31449.log 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.69782.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36587.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51362.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45494.log 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.69782.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62426.log 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.31449.log: /var/log/ceph/cluster2-client.mirror.45494.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31449.log.gz 2024-09-16T11:45:40.317 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53630.log 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62426.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45494.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63692.log 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.admin.53630.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52376.log 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62426.log.gz 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.53630.log.gz 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.63692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54056.log 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.52376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54965.log 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52376.log.gz 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.54056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29699.log 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63692.log.gz/var/log/ceph/cluster2-client.admin.54965.log: 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61204.log 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.54965.log.gz 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.54056.log.gz 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29699.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.29699.log.gz 2024-09-16T11:45:40.318 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31347.log 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.61204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42392.log 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.31347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61382.log 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr: 2.3% 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.61204.log.gz 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.31347.log.gz 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57312.log 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.42392.log: /var/log/ceph/cluster2-client.mirror.61382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53539.log 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57312.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42392.log.gz 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61382.log.gz 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.57312.log.gz 2024-09-16T11:45:40.319 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63902.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.53539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49771.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.63902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61555.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.53539.log.gz 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63902.log.gz 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.49771.log: -- /var/log/ceph/cluster2-client.mirror.62592.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49397.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.49771.log.gz 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61555.log.gz 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.62592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40456.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.49397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67391.log 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62592.log.gz 2024-09-16T11:45:40.320 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0%/var/log/ceph/cluster1-client.admin.40456.log: -- replaced with /var/log/ceph/cluster2-client.admin.49397.log.gz 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68184.log 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37663.log 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.40456.log.gz/var/log/ceph/cluster1-client.mirror.68184.log: 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43314.log 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.68184.log.gz 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54726.log 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37663.log: /var/log/ceph/cluster1-client.mirror.43314.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67391.log.gz 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.37663.log.gz 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38124.log 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9%/var/log/ceph/cluster2-client.admin.54726.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43314.log.gz 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40051.log 2024-09-16T11:45:40.321 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32928.log 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.54726.log.gz 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38124.log: /var/log/ceph/cluster2-client.admin.40051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71485.log 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38124.log.gz 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.32928.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.40051.log.gz 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54993.log 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66210.log 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.admin.32928.log.gz 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71485.log.gz 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.54993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61842.log 2024-09-16T11:45:40.322 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.66210.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54993.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30128.log 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61610.log 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61842.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30128.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30128.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.42333.log 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.61610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56086.log 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66210.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.32525.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61610.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.admin.42333.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.59328.log 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.56086.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49456.log 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.42333.log.gz 2024-09-16T11:45:40.323 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56086.log.gz 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.59328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48807.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.49456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69297.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr: 90.2%/var/log/ceph/cluster1-client.mirror.48807.log: -- replaced with /var/log/ceph/cluster1-client.admin.59328.log.gz 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71967.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49456.log.gz 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.69297.log: -- /var/log/ceph/cluster2-client.admin.40959.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71967.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48807.log.gz 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.69297.log.gz 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.55402.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36917.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.40959.log.gz/var/log/ceph/cluster2-client.admin.55402.log: 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52182.log 2024-09-16T11:45:40.324 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.36917.log: -- replaced with /var/log/ceph/cluster2-client.admin.55402.log.gz 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35040.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71967.log.gz 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67700.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69480.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.35040.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.52182.log.gz 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49249.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.67700.log: 92.6% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36917.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.35040.log.gz 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.69480.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67700.log.gz 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61241.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.49249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36822.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61241.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49249.log.gzgzip 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.29618.log 2024-09-16T11:45:40.325 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39205.log 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69480.log.gz 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29618.log: /var/log/ceph/cluster2-client.admin.36822.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29618.log.gz 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67233.log 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.39205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37292.log 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61241.log.gz 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36822.log.gz/var/log/ceph/cluster2-client.mirror.67233.log: 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67233.log.gz 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.39088.log 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37292.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36519.log 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.37292.log.gz/var/log/ceph/cluster1-client.admin.39088.log: 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34362.log 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39205.log.gz 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.39088.log.gz 2024-09-16T11:45:40.326 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42304.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58705.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34362.log: gzip -5 --verbose/var/log/ceph/cluster1-client.admin.42304.log: -- /var/log/ceph/cluster2-client.mirror.42876.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.admin.34362.log.gz 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42304.log.gz 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.58705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34331.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.42876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71335.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.34331.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42876.log.gz 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34331.log.gz 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55164.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.71335.log: 90.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56179.log 2024-09-16T11:45:40.327 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58705.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71335.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35873.log 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55164.log: /var/log/ceph/cluster1-client.mirror.56179.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35873.log: 87.4% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.55164.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56179.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35873.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60311.log 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48748.log 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2.log: gzip/var/log/ceph/cluster2-client.mirror.60311.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.55131.log 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.48748.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60311.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.48748.log.gz 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66713.log 2024-09-16T11:45:40.328 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.55131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39322.log 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.66713.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.55131.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66713.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56273.log 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr: 87.1% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.39322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73558.log 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36519.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6%/var/log/ceph/cluster2-client.admin.56273.log: -- replaced with /var/log/ceph/cluster1-client.admin.39322.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37387.log 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.56273.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73558.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73558.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr: -- /var/log/ceph/cluster1-client.mirror.38012.log 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30254.log 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38012.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.37387.log.gz 2024-09-16T11:45:40.329 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52997.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38012.log.gz 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.30254.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30254.log.gz 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70140.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.52997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52809.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.70140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41895.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70140.log.gz 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52997.log.gz 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.52809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36959.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52809.log.gz 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71171.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61501.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.41895.log.gz 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36959.log: /var/log/ceph/cluster2-client.mirror.71171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60093.log 2024-09-16T11:45:40.330 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56699.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.60093.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71171.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36959.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31654.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61501.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.56699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39481.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53846.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56699.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.31654.log: /var/log/ceph/cluster1-client.mirror.39481.log: 53.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40549.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.31654.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3%/var/log/ceph/cluster1-client.admin.53846.log: -- replaced with /var/log/ceph/cluster1-client.mirror.39481.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35509.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.60093.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.53846.log.gz 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49277.log 2024-09-16T11:45:40.331 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.40549.log: /var/log/ceph/cluster1-client.mirror.35509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62284.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35509.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.49277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48368.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.40549.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.62284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61009.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62284.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33531.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49277.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.48368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33010.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.61009.log: /var/log/ceph/cluster1-client.admin.33531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38089.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.48368.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61009.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.33531.log.gz 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose --/var/log/ceph/cluster2-client.mirror.33010.log: /var/log/ceph/cluster1-client.admin.38538.log 2024-09-16T11:45:40.332 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.38089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32304.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38089.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.38538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42479.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43712.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.38538.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.32304.log: /var/log/ceph/cluster1-client.admin.42479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67633.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.32304.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.43712.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42479.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44801.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.43712.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57487.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.67633.log: /var/log/ceph/cluster1-client.admin.44801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37740.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67633.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.57487.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44801.log.gz 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63174.log 2024-09-16T11:45:40.333 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.57487.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72021.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.37740.log: /var/log/ceph/cluster2-client.mirror.63174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41397.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.72021.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63174.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.admin.72021.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29674.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37740.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41833.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29674.log: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34506.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29674.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.41397.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30674.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.41833.log: /var/log/ceph/cluster1-client.admin.34506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36070.log 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.30674.log: 0.0% 88.4% -- replaced with /var/log/ceph/cluster2-client.admin.41833.log.gz 2024-09-16T11:45:40.334 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.30674.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34506.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52738.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45179.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.36070.log: /var/log/ceph/cluster2-client.mirror.52738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37154.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33010.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.45179.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52738.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39021.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.37154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37236.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.39021.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.admin.37154.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45179.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39021.log.gz 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40139.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.37236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37323.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.40139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73976.log 2024-09-16T11:45:40.335 INFO:teuthology.orchestra.run.smithi204.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37236.log.gz 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.37323.log: 90.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51755.log 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40139.log.gz 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.37323.log.gz 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:gzip/var/log/ceph/cluster2-client.admin.73976.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37415.log 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73976.log.gz 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52408.log 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.37415.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.51755.log.gz 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73071.log 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose 89.1% -- /var/log/ceph/cluster2-client.mirror.58250.log 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.52408.log: -- replaced with /var/log/ceph/cluster2-client.mirror.37415.log.gz 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35261.log 2024-09-16T11:45:40.336 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73071.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.52408.log.gz 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73071.log.gz 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67893.log 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58250.log.gz 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29086.log 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56453.log 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.35261.log.gz/var/log/ceph/cluster2-client.mirror.67893.log: 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.29086.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29086.log.gz 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67893.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39516.log 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.56453.log: /var/log/ceph/cluster2-client.admin.73474.log 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.39516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53314.log 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.56453.log.gz 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39516.log.gz 2024-09-16T11:45:40.337 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.73474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71768.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73474.log.gz 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.53314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43055.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56912.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.53314.log.gz 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.71768.log: /var/log/ceph/cluster1-client.mirror.43055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35584.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71768.log.gz 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.56912.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43055.log.gz 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62678.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.56912.log.gz 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52341.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.35584.log: /var/log/ceph/cluster1-client.mirror.62678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64556.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35584.log.gz 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.52341.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62678.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63344.log 2024-09-16T11:45:40.338 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52341.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57580.log 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.64556.log: /var/log/ceph/cluster2-client.mirror.63344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30716.log 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.57580.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57580.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51725.log 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.30716.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63344.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30716.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64556.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.admin.51725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54567.log 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:gzip 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.51725.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65786.log 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.54567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41926.log 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.65786.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54567.log.gz 2024-09-16T11:45:40.339 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71582.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32211.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65786.log.gz/var/log/ceph/cluster2-client.mirror.71582.log: 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.41926.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71582.log.gz 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54905.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41926.log.gz 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.32211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32381.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.admin.54905.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.32211.log.gz 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52151.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.54905.log.gz 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70598.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.32381.log: /var/log/ceph/cluster1-client.admin.52151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36680.log 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.70598.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.32381.log.gz 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.52151.log.gz 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70598.log.gz 2024-09-16T11:45:40.340 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58107.log 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.36680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60404.log 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36680.log.gz 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster2-client.mirror.58107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53117.log 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58107.log.gz 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53247.log 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/cluster1-client.mirror.60404.log: /var/log/ceph/cluster2-client.admin.53117.log: /var/log/ceph/cluster1-client.mirror.53247.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.admin.53117.log.gz 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53247.log.gz 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60404.log.gz 2024-09-16T11:45:40.341 INFO:teuthology.orchestra.run.smithi204.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36070.log.gz 2024-09-16T11:45:40.577 INFO:teuthology.orchestra.run.smithi204.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-16T11:45:42.441 INFO:teuthology.orchestra.run.smithi053.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-16T11:45:42.996 INFO:teuthology.orchestra.run.smithi204.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-16T11:45:43.176 INFO:teuthology.orchestra.run.smithi053.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-16T11:45:43.360 INFO:teuthology.orchestra.run.smithi204.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-16T11:45:43.508 INFO:teuthology.orchestra.run.smithi053.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-16T11:45:43.510 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-09-16T11:45:43.510 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m3.990s 2024-09-16T11:45:43.510 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m11.204s 2024-09-16T11:45:43.510 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.388s 2024-09-16T11:45:44.097 INFO:teuthology.orchestra.run.smithi204.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-16T11:45:44.099 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-16T11:45:44.099 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m4.548s 2024-09-16T11:45:44.100 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m12.918s 2024-09-16T11:45:44.100 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m1.347s 2024-09-16T11:45:44.100 INFO:tasks.ceph:Archiving logs... 2024-09-16T11:45:44.100 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/remote/smithi053/log 2024-09-16T11:45:44.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T11:45:45.621 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/remote/smithi204/log 2024-09-16T11:45:45.622 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T11:45:48.377 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T11:45:48.390 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T11:45:48.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:45:48.393 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:45:48.475 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T11:45:48.484 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T11:45:48.484 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:45:48.487 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:45:48.503 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:45:48.503 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-16T11:45:48.508 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:45:48.508 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-09-16T11:45:48.508 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 3 6 377 17 -476us[ -485us] +/- 37ms 2024-09-16T11:45:48.508 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 +1369us[+1360us] +/- 63ms 2024-09-16T11:45:48.508 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -661us[ -670us] +/- 65ms 2024-09-16T11:45:48.508 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:45:48.509 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T11:45:48.509 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-16T11:45:48.510 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 -481us[ -484us] +/- 37ms 2024-09-16T11:45:48.510 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 +1379us[+1379us] +/- 63ms 2024-09-16T11:45:48.510 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -672us[ -672us] +/- 65ms 2024-09-16T11:45:48.510 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T11:45:48.510 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T11:45:48.556 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T11:45:48.557 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T11:45:48.573 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T11:45:48.645 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T11:45:48.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:45:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:48.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T11:45:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:48.776 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T11:45:48.786 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T11:45:48.794 INFO:teuthology.task.internal:Duration was 1805.456745 seconds 2024-09-16T11:45:48.795 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T11:45:48.803 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T11:45:48.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:45:48.806 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:45:48.846 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:45:48.857 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T11:45:49.076 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T11:45:49.076 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-09-16T11:45:49.077 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:45:49.138 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-09-16T11:45:49.138 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:45:49.160 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T11:45:49.160 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:45:49.180 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:45:49.228 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T11:45:49.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:45:49.248 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:45:49.373 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T11:45:49.383 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T11:45:49.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:45:49.409 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:45:49.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T11:45:49.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:49.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:49.474 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-09-16T11:45:49.498 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-09-16T11:45:49.618 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:49.675 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:49.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T11:45:49.700 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T11:45:49.701 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/remote/smithi053 2024-09-16T11:45:49.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:45:49.750 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907185/remote/smithi204 2024-09-16T11:45:49.750 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:45:49.783 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T11:45:49.784 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:45:49.787 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:45:49.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T11:45:49.845 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T11:45:49.845 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T11:45:49.855 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T11:45:49.855 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:45:49.857 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:45:49.870 INFO:teuthology.orchestra.run.smithi053.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:45 /home/ubuntu/cephtest 2024-09-16T11:45:49.889 INFO:teuthology.orchestra.run.smithi204.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:45 /home/ubuntu/cephtest 2024-09-16T11:45:49.890 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T11:45:49.900 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T11:45:49.925 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_latest} workloads/rbd-mirror-workunit-min-compat-client-octopus} duration: 1805.456744670868 flavor: default owner: scheduled_sangadi@teuthology success: true 2024-09-16T11:45:49.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T11:45:50.009 INFO:teuthology.run:pass